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HEARING CHARTER 
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Purpose 

The purpose of this hearing is to inform the Subcommittee of educators’ experi- 
ences working with science, technology, engineering, and math (STEM) education 
programs for K-16 students supported by the federal R&D mission agencies: Na- 
tional Aeronautics & Space Administration (NASA), National Oceanographic & At- 
mospheric Administration (NOAA), National Institute of Standards & Technology 
(NIST), Environmental Protection Agency (EPA), and Department of Energy (DOE). 
This hearing will explore whether such issues as the lack of coordination between 
the agencies, difficulty by educators in finding information about the programs, and 
the absence of robust evaluation techniques hinder the potential of the federal pro- 
grams for improving STEM education in America. Most importantly, the hearing 
will highlight how the federal R&D mission agencies can best contribute to raising 
the level of scientific literacy of all students. 

Witnesses 

Ms. Linda K. Froschauer, President, National Science Teachers Association 
Mr. Michael C. Lach, Director of Mathematics and Science, Chicago Public Schools 

Dr. George D. Nelson, Director, Science, Technology, and Mathematics Education, 
Western Washington University 

Mr. Van R. Reiner, President, Maryland Science Center 
Dr. Iris R. Weiss, President, Horizon Research, Inc. 

Overarching Questions 

• What are the experiences of educators in finding and leveraging resources for 
STEM education from the federal R&D mission agencies? What challenges 
have they encountered? 

• What do educators perceive to be successful STEM education programs at the 
federal R&D mission agencies? How do they determine success? What should 
the agencies improve? 

• What support that the federal R&D mission agencies could provide would 
have the most impact on improving STEM education? 

Background 

A multitude of studies over the past twenty years have documented the downward 
slide of American students’ proficiency and participation in science, technology, engi- 
neering and mathematics (STEM) fields. In October 2005, the National Academies 
released the report, Rising Above the Gathering Storm: Energizing and Employing 
America for a Brighter Economic Future, which warned that “the scientific and tech- 
nological building blocks critical to our economic leadership are eroding at a time 
when many other nations are gathering strength.” The authoring committee deemed 
the highest priority action to be vast improvement of science and mathematics edu- 
cation in order to increase the number of students interested in and prepared for 
entering careers in STEM fields. 

The scientific and technical expertise of the R&D mission agencies has been ap- 
plied to varying degrees and with varying success to programs and activities rel- 
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evant to improving STEM education. Currently, the STEM education programs at 
these agencies for K-16 grades are relatively small and vary widely in methods, tar- 
get audiences, evaluation measurements, and funding. In an inventory of STEM 
education programs, the Academic Competitiveness Council (ACC) identified ap- 
proximately $39 million in FY06 appropriated funds at the federal R&D mission 
agencies for K-12 programs and $6.6 million for undergraduate programs. 

Until recently, the agencies have developed their programs independently and 
without a strategic plan for accomplishing a set of overarching goals and objectives. 
Unfortunately, this led to a need for each program to discover a cadre of “best prac- 
tices” on its own rather than collaborating with other program and agency experts. 
Each program also developed a unique method of evaluation, making a comparison 
of effectiveness across the programs impossible. Lastly, the agencies have had trou- 
ble building widespread awareness of their programs among teachers nationwide. 

In response to these issues, the National Science and Technology Council, which 
serves as the principal body for coordinating federal research and development, has 
re-established the Education and Workforce Development subcommittee to encour- 
age the agencies to share knowledge and develop a federal strategic plan for effec- 
tively increasing STEM proficiency nationwide. The plans for this new federal co- 
ordination and planning activity will be reviewed by the Subcommittee in a subse- 
quent hearing. 

The Subcommittee recognizes that critical guidance for how to improve the federal 
STEM education programs must come from the people who work directly with 
teachers and students. The witnesses for today’s hearing were chosen because of 
their experience working with these programs and have been asked to provide in- 
sight from the field on what the agencies are doing well and where they need to 
improve. 

Specific Questions for the Witnesses 

Ms. Linda K. Froschauer 

• In your experience, what are the federal R&D mission agencies doing well in 
their respective STEM education programs? What could they do better? Can 
you give examples of any particularly effective programs? 

• How do your teacher members learn about STEM education programs spon- 
sored by the federal research and development agencies? 

• What resources of the agencies would be most valuable in supporting your 
teacher members in the classroom? 

Mr. Michael C. Lack 

• How do you find resources for improving science and mathematics education 
in the Chicago Public Schools? 

• What resources have you garnered from the federal R&D mission agencies? 
How has this contributed to improving your students’ understanding of 
science? 

• What type of support that the federal R&D mission agencies could provide 
would have the most impact on STEM education for your teachers and stu- 
dents in Chicago Public Schools? 

Dr. George D. Nelson 

• In what ways can federal R&D mission agencies contribute most effectively 
to improve K-12 STEM education? Can you give examples of any particularly 
effective programs? 

• At the undergraduate level, what type of support could the federal R&D mis- 
sion agencies provide that would recruit more students into pursuing careers 
in the physical sciences? 

• How does the lack of coordination and overarching strategy for STEM edu- 
cation programs hinder the agencies from making an impact? 

Mr. Van R. Reiner 

• Please describe the informal education programs that you have partnered 
with federal R&D mission agencies to provide for school-aged children. 

• How well do the federal R&D mission agencies develop evaluation methods 
to determine the effectiveness of informal STEM education programs? 
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• What informal education activities should the federal R&D mission agencies 
increase to help raise the level of scientific literacy in American students? 

Dr. Iris R. Weiss 

• Do you feel that the federal R&D mission agencies develop evaluation meth- 
ods for STEM education programs that demonstrate effectiveness? What rec- 
ommendations would you give for improving their evaluation methods? 

• To what extent do the federal R&D mission agencies incorporate best prac- 
tices which have proven to be effective in STEM education into their pro- 
grams? 

• Based on your research on teacher training and professional development, 
what guidance would you give for developing programs for pre-service and in- 
service STEM teachers? 
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Chairman Baird. This hearing will come to order. I want to 
thank the witnesses and our guests and thank my good friend and 
colleague, Dr. Ehlers, Ranking Member. 

Our Subcommittee on Research and Science Education Com- 
mittee is interested in hearing from educators in science, tech- 
nology, engineering and mathematics, STEM fields, about their ex- 
periences working with the federal R&D mission agencies. 

This hearing is part of the ongoing effort led by Chairman Gor- 
don that the Committee is undertaking to determine how to im- 
prove the level of scientific understanding of students in the United 
States and how to attract more students to careers in science and 
engineering. 

There have been at least a half-dozen reports released over the 
past 20 years documenting how American students have fallen be- 
hind students in other countries. The National Academies report, 
Rising Above the Gathering Storm, warned us that this will threat- 
en the standing of our country in the future. The authors of that 
paper wrote “the scientific and technological building blocks critical 
to our economic leadership are eroding at a time when many other 
nations are gathering strength.” They recommended, they being the 
authors, recommended that the highest priority should be a vast 
improvement of science and mathematics education in this country 
in order to increase the number of students interested in and pre- 
pared to enter careers in STEM fields. 

The Science and Technology Committee held a hearing in March 
with leading voices in private industry and higher education to dis- 
cuss research and education needs in STEM fields. Every one of 
the witnesses, including a retired CEO of Lockheed Martin, the 
current CEO of McGraw-Hill, the CEO of Intel, and the President 
of the Council on Competitiveness, testified that companies in 
America need a workforce well-trained in STEM fields in order to 
continue the innovative solutions that keep them profitable. 

The Committee has taken this advice to heart. H.R. 362, also 
known as “10,000 Teachers, 10 Million Minds” Science and Math 
Scholarship Act, was introduced by Chairman Gordon earlier this 
year. The bill implements most of the K-12 education recommenda- 
tions of the Gathering Storm report and was passed by the House 
with strong bipartisan support last month. 

The Research and Science Education Subcommittee will next be 
exploring ways that federal efforts in STEM education can be bet- 
ter focused and more effective. This is the first in a planned series 
of hearings to address these issues. 

Today we are reviewing the role of the federal R&D mission 
agencies in improving STEM education. Specifically, we are refer- 
ring to NASA, NOAA, NIST, EPA, and the Department of Energy. 
I believe there is a great deal of untapped potential residing in the 
expertise of scientists and engineers at these agencies. 

Not only do these scientists and engineers possess impressive 
content knowledge of the sciences, they also have real-world experi- 
ence with the “wow” factor that gets kids excited about learning 
science. Space travel, discovering new forms of ocean life, creating 
renewable energy sources, improving air and water quality, testing 
bullet-proof vests, we could list hundreds of more activities that 
make science and math captivating to young people. 
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Although the agencies have made commendable efforts to share 
their knowledge and passion for science with students, I believe 
those efforts have been relatively small and have varied widely in 
their methods, target audiences, and methods of evaluation. The 
programs have been developed independently and without a stra- 
tegic plan for accomplishing a common set of goals and objectives. 
With a unified effort, I am convinced these programs could have a 
much bigger impact on the approximately 52 million K-12 students 
in America. 

So we have asked our witnesses today to tell us about their expe- 
riences participating in these programs. We have asked them to re- 
spond to a series of questions. What do the agencies do well? What 
should they improve? Which programs do educators consider suc- 
cessful? And how do they define that success? 

The Committee is devoted to improving science education, so de- 
voted that we added science education to the name of this sub- 
committee. We are very concerned that American students are not 
achieving their potential in science and math education. It is a con- 
cern not only as we look at competing in a knowledge-based global 
economy but also when we look at access to high-paying, tech- 
nology-based jobs in this country. 

I look forward to hearing from today’s witnesses and recognize 
my colleague and Ranking Member, Dr. Ehlers. 

[The prepared statement of Chairman Baird follows:] 

Prepared Statement of Chairman Brian Baird 

Good morning and thank you for attending today’s Subcommittee on Research and 
Science Education hearing. Today, we are going to hear from educators in science, 
technology, engineering, and mathematics — STEM fields — about their experiences 
working with the federal R&D mission agencies. 

This hearing is part of an ongoing effort that the Committee is undertaking to 
determine how to improve the level of scientific understanding of students in the 
U.S. and how to attract more students to careers in science and engineering. 

There have been at least a half dozen reports released over the past 20 years doc- 
umenting how American students have fallen behind students in other countries. 
The National Academies report, Rising Above the Gathering Storm, warned us that 
this will threaten the standing of our country in the future. The authors wrote “the 
scientific and technological building blocks critical to our economic leadership are 
eroding at a time when many other nations are gathering strength.” They rec- 
ommended that the highest priority should be a vast improvement of science and 
mathematics education in this country in order to increase the number of students 
interested in and prepared to enter careers in STEM fields. 

The Science and Technology Committee held a hearing in March with leading 
voices in private industry and higher education to discuss research and education 
needs in STEM fields. Every one of the witnesses, including a retired CEO of Lock- 
heed Martin, the current CEO of McGraw-Hill, the CEO of Intel, and the President 
of the Council on Competitiveness, testified that companies in America need a work- 
force well-trained in STEM fields in order to continue the innovative solutions that 
keep them profitable. 

The Committee has taken this advice to heart. H.R. 362, “10,000 Teachers, 10 Mil- 
lions Minds” Science and Math Scholarship Act, was introduced by Chairman Gor- 
don early this year. The bill implements most of the K-12 education recommenda- 
tions of the Gathering Storm report and was passed by the House last month. 

The Research and Science Education Subcommittee will next be exploring ways 
that federal efforts in STEM education can be better focused and more effective. 
This is the first in a planned series of hearings to address these issues. 

Today we are reviewing the role of the federal R&D mission agencies in improving 
STEM education. Specifically, I am referring to NASA, NOAA, NIST, EPA, and the 
Department of Energy. I believe there is a great deal of untapped potential residing 
in the expertise of scientists and engineers at these agencies. 
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Not only do these scientists and engineers possess impressive content knowledge 
in the sciences, they also have real-world experience with the “wow” factors that 
gets kids excited about learning science. Space travel, discovering new forms of 
ocean life, creating renewable energy sources, improving air and water quality, test- 
ing bullet-proof vests — I could list hundreds more activities that make science and 
math captivating. 

Although these agencies have made commendable efforts to share their knowledge 
and passion for science with students, I fear that those efforts have been relatively 
small and have varied widely in their methods, target audiences, and methods of 
evaluation. The programs have been developed independently and without a stra- 
tegic plan for accomplishing a common set of goals and objectives. With a unified 
effort, I am convinced that these programs could have a much bigger impact on the 
approximately 52 million K-12 students in America. 

We have asked our witnesses today to tell us about their experiences participating 
in these programs. We have asked them to respond to a series of questions: What 
do the agencies do well? What should they improve? Which programs do educators 
consider successful? And how do they define that success? 

The Committee is devoted to improving science education — so devoted that we 
added science education to the name of this subcommittee. We are very concerned 
that American students are not achieving their potential in science and math edu- 
cation. This is a concern not only as we look at competing in a knowledge-based 
global economy, but also when we look at access to high-paying, technology-based 
jobs in this country. 

I look forward to hearing from today’s witnesses. 

Mr. Ehlers. Thank you very much, Chairman Baird, and I do 
apologize to you and to the group for my due late arrival. I was 
speaking at another session, and unfortunately I ended up being 
the last speaker; and even though it was difficult for me, I did cut 
my words short. 

I am pleased today that we have a cadre of consumers of science, 
technology, engineering and mathematics, better known as STEM 
educational programs across the federal agencies before us to hear 
about their experiences. 

I believe it is the desire of all Members of the Science and Tech- 
nology Committee that we support the implementation of programs 
that are well-designed and effective. But often as legislators we are 
so distanced from the final implementation of the programs that we 
hear little about personal challenges and personal successes, and 
this is an opportunity for us to hear what some of our grand 
schemes have resulted in and hear it from the people who are sort 
of the boots on the ground in the STEM education battle. 

Today’s hearing will delve into what was happening with the 
consumers of federal STEM agency programs. I might mention 
many of these programs don’t come directly from this committee. 
There are a number of federal agencies that instigate their own 
programs without Congressional direction, and I have worried for 
some years how those all intertwine with each other and with what 
we have passed. 

While each of our panelists brings unique perspectives to the 
table today, I note there are a few common themes running 
through your prepared testimony. Several of you have identified 
the Federal Science and Technology Workforce and Facilities and 
under-utilized resources for K-12 classrooms. I am interested to 
learn more about programs that successfully leverage these re- 
sources. 

Secondly, many of you remarked that the best programs are 
those that excite and inform teachers and students. Finally, your 
testimony, coupled with the recent release of the Academic Com- 
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petitiveness Council’s report on federal STEM programs empha- 
sizes a need to reduce the number of programs that are not evalu- 
ated or clearly do not provide a benefit to teachers and students. 

Alternatively, faced with a maze of resources, you need help 
identifying programs that have been evaluated as successful to 
know what may be useful to you. 

Our challenge in Congress is to target limited federal funds at 
programs which leverage relevant federal resources and also com- 
plement the local education requirement. 

Today we will have achieved a win-win scenario to promote 
STEM literacy at all levels if we manage to do that. I am particu- 
larly pleased to see that today’s panel includes Michael Lach. As 
an Einstein Fellow in my office from 1999 to 2000, Michael pro- 
vided extremely valuable insight to me on STEM education and 
science policy. He has moved onto much grander things, now di- 
recting the math and science curricula for the entire Chicago Pub- 
lic School System. He has been an outspoken pioneer for effective 
teaching in math and science from the time that he started teach- 
ing high school science through the Teach for America program. 

Granted that I will allow that I am a little biased since Michael 
is a physicist by training, but I have been told by others that he 
is an exceptional teacher, one that other teachers look up to as an 
example that they aspire to be. We are fortunate to have him here 
today. Welcome Michael, and for that I yield back. 

[The prepared statement of Mr. Ehlers follows:] 

Prepared Statement of Representative Vernon J. Ehlers 

I am pleased today that we have a cadre of consumers of Science, Technology, En- 
gineering and Mathematics (STEM) educational programs across the federal agen- 
cies before us to hear about their experiences. I believe it is the desire of all the 
Members of the Science and Technology Committee that we support the implemen- 
tation of programs that are well-designed and effective. Often as legislators we are 
so distanced from final implementation of programs that we hear little about per- 
sonal challenges and successes. Today’s hearing will delve into what is happening 
with the consumers of federal STEM agency programs. 

While each of our panelists brings unique perspectives to the table today, I note 
that there are a few common themes running through your prepared testimony. 
Several of you have identified the federal science and technology workforce and fa- 
cilities as under-utilized resources for K-16 classrooms. I am interested to learn 
more about programs that successfully leverage these resources. Secondly, many of 
you remark that the best programs are those that excite and inform students and 
teachers. Finally, your testimony — coupled with the recent release of the Academic 
Competitiveness Council’s (ACC) report on federal STEM programs — emphasizes a 
need to reduce the number of programs that are not evaluated or clearly do not pro- 
vide a benefit to teachers and students. Alternatively, faced with a maze of re- 
sources, you need help identifying programs that have been evaluated as successful 
to know what may be useful to you. Our challenge in Congress is to target limited 
federal funds at programs which leverage relevant federal resources and also com- 
plement the local educational requirements. Then we will have achieved a win-win 
scenario to promote STEM literacy at all levels. 

I am particularly pleased to see that today’s panel includes Michael Lach. As an 
Einstein Fellow in my office from 1999-2000, Michael provided extremely valuable 
insight to me on STEM education and science policy. He has moved on to much 
grander things, now directing the science and math curricula for the entire Chicago 
Public School system. He has been an outspoken pioneer for effective teaching in 
math and science from the time that he started teaching high school science through 
the Teach for America program. Granted, I will allow that I am a little biased since 
Michael is a physicist by training, but I have been told by others that he is an ex- 
ceptional teacher, one that other teachers look up to as the example they aspire to 
be. We are fortunate to have him here with us today. Welcome, Michael. 
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Chairman Baird. Thank you, Dr. Ehlers. If there are other Mem- 
bers who wish to submit additional opening statements, the state- 
ments will be added to the record. 

[The prepared statement of Ms. Johnson follows:] 

Prepared Statement of Representative Eddie Bernice Johnson 

Thank you, Mr. Chairman. Our nation’s future competitiveness depends on 
whether or not tomorrow’s generation is prepared for the high-tech jobs of the fu- 
ture. 

Enthusiastic, adequately prepared teachers who utilize successful teaching meth- 
ods are needed in many of today’s math and science classrooms. 

Mr. Chairman, the Federal Government must invest in math and science edu- 
cation, as these investments help fuel our nation’s economic growth. 

Economists agree that no other investment generates a greater long-term return 
to the economy than scientific R&D, and that starts with educational systems. Re- 
search, education, the technical workforce, scientific discovery, innovation and eco- 
nomic growth are intertwined. 

The Science, Technology, Engineering and Math Caucus has reported that Texas 
ranked 20th in the Nation on the 2005 National Assessment of Educational Progress 
scores for mathematics with a score of 281. The national average was 278. 

Texas did not report on the percentage of Texas middle school teachers who were 
certified in math (the national average was 49 percent) or science (the national av- 
erage was 54 percent). 

Sadly, only seven percent of Texas’ 12th grade students took the AP Calculus 
exam in 2004. Students should be challenged so that they are able to master these 
subjects, if we want to compete globally. 

Today’s witnesses are here to provide a critical view of federal STEM education 
programs. Members of Congress need to know what works and what doesn’t work 
so that we can build policies that offer the most benefit per taxpayer dollar. 

Thank you, Mr. Chairman. I yield back. 

Chairman Baird. At this time I would like to introduce the wit- 
nesses on the panel. Ms. Linda Froschauer is the President of Na- 
tional Science Teachers Association. She is also the K-12 Science 
Department Chair for the Weston Public Schools in Weston, Con- 
necticut. It is good to have you with us again. We always value 
your remarks and insights. Mr. Michael Lach is the Director of 
Mathematics and Science for Chicago Public Schools. Dr. George D. 
Nelson is Director of the Science, Mathematics, and Technology 
Education and is Professor of Physics and Astronomy at Western 
Washington University, my home state. He is a former astronaut 
and flew on the Space Shuttle. Mr. Van Reiner is the President of 
the Maryland Science Center and formerly was the President of the 
Sparrow Point Division of Bethlehem Steel. And Dr. Iris Weiss is 
President of Horizon Research, Incorporated, which specializes in 
mathematics and science education and research evaluation. As 
you can see, we have a very distinguished and well-qualified panel 
before us today to learn from. 

I would remind our witnesses — first of all, we have all had a 
chance to look at your written testimony. Thank you for preparing 
that, some of it quite lengthy, but very, very informative; but I 
would remind you that today for the purpose of testimony — spoken 
testimony is limited to five minutes each. There are little lights 
there that will come on; and as Dr. Ehlers used to remind people, 
after the red light comes on, you have about five seconds and a 
trapdoor appears underneath your chair and you disappear from 
sight, after which Members of the Subcommittee will have five 
minutes each to ask questions. This is a collegial, friendly atmos- 
phere, so we very much look forward to a good exchange of ideas. 



11 


And with that, we will start with Ms. Froschauer. 

STATEMENT OF MS. LINDA K. FROSCHAUER, PRESIDENT, 
NATIONAL SCIENCE TEACHERS ASSOCIATION 

Ms. Froschauer. Thank you for this opportunity to present tes- 
timony on behalf of the National Science Teachers Association. 

My name is Linda Froschauer, and I am the President of the 
NSTA. For 32 years I have been a science teacher, currently teach- 
ing eighth-grade science, and I am Department Chair at Weston 
Middle School in Connecticut. 

This is the second opportunity I have had in recent months to 
testify before this subcommittee. A few weeks ago I appeared in 
support of H.R. 524 which is for Partnerships for Access to Labora- 
tory Science Grants. I want to sincerely thank the Members of the 
Science Committee and the House for passing this extremely im- 
portant legislation as part of H.R. 362. 

Today I would like to talk about federal STEM education pro- 
grams for K-12 teachers. As you know, the vast majority of STEM 
education programs are generated from the U.S. Department of 
Education and the National Science Foundation. We consider the 
NSF to be the engine of innovation for K-12 STEM education. In- 
formation about NSF initiatives and other federal STEM education 
programs are promoted extensively through NSTA print and online 
channels and on the NSTA website. Combined, these communica- 
tion vehicles reach hundreds of thousands of teachers, teacher lead- 
ers, and others in the science education community. 

Federal agencies also share information about programs for 
science educators at the NSTA conferences which draw approxi- 
mately 12,000 teachers each year. 

During the last NSTA conference, the National Institutes of 
Health featured the NIH Research Zone, a coordinated effort that 
involved 27 institutes and centers from NIH professional societies 
and other supporting partners. The NIH Research Zone provided 
one-stop shopping for teachers interested in discovering the re- 
sources available from the NIH research community. Workshops 
and exhibits on NASA’s education programs are also prominent 
parts of NSTA conferences. These include the NASA Educator As- 
tronaut Launch where teachers can join NASA’s first educator as- 
tronaut, Barbara Morgan, on her upcoming Shuttle launch. The 
Student Observation Network, 21st Century Explorer, and the En- 
gineering Design Challenge allow students to use NASA data to 
conduct their own analysis and apply engineering principles to 
solve scientific problems. 

One of the challenges with many federal education programs, 
however, is that they reach only a miniscule proportion of our na- 
tion’s science teachers. We must continue to find new and effective 
ways to get quality, professional development programs up to scale 
so they reach a large number of teachers. 

To address this issue, NSTA is working with NASA, NOAA, and 
the FDA to develop face-to-face training and online experiences 
that we believe have potential to reach hundreds of thousands of 
K-12 teachers. Why is professional development so important? Last 
year the National Research Council report titled Taking Science to 
School, Learning and Teaching Science in Grades K-8, said that 
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professional development is key to supporting effective science in- 
struction in the critical early years of a child’s education. All teach- 
ers need opportunities to deepen their knowledge of the science 
content. In fact, the NRC and Taking Science to School Report says 
that federal agencies that support professional development should 
require that the programs they fund incorporate models of instruc- 
tion that combine the four strands of science proficiency, focus on 
core ideas in science, and enhance teacher’s science content knowl- 
edge, knowledge of how students learn science, and knowledge of 
how to teach science. 

In conclusion, recent reports have made it clear that better co- 
ordination and communication is desperately needed among federal 
agencies, bureaus, divisions, and centers that are involved with 
STEM education research and programs. The federal agencies do 
not appear to work together to facilitate the dissemination of re- 
search or to discuss possible new ideas and avoid those duplicative 
programs. In addition, an inventory of STEM education programs 
across the federal agencies would inform future priorities and ini- 
tiatives, an oversight entity at the federal level that works to co- 
ordinate STEM education programs and could work with state and 
local officials and with science education stakeholders is critical. 
Improvements in STEM education require a commitment of leader- 
ship at the local, State, and federal levels. Education programs at 
the federal agencies should continue to play a role in improving 
STEM education. 

We hope that any changes to existing programs, especially at the 
National Science Foundation, that may come about as a result of 
this ACC report will be carefully reviewed and considered. Thank 
you for allowing me the opportunity to address you today, and I 
look forward to your questions. 

[The prepared statement of Ms. Froschauer follows:] 

Prepared Statement of Linda K. Froschauer 
Mr. Chairman and Members of the Committee: 

Thank you for this opportunity to present testimony on behalf of the National 
Science Teachers Association. My name is Linda Froschauer, and I am President of 
the NSTA. For 32 years I have been a science teacher and I am currently an 8th 
grade science teacher and Department Chair at the Weston Public Schools in Con- 
necticut. 

This is the second opportunity I have had in recent months to testify before this 
subcommittee. A few months ago I appeared in support of H.R. 524, the Partner- 
ships for Access to Laboratory Science grants. I want to sincerely thank the mem- 
bers of the Science Committee and the House for passing this extremely important 
legislation as part of H.R. 362. 

Today I will talk about the Federal STEM education programs for K-12 teachers. 

As you know the vast majority of STEM education programs originate from the 
U.S. Department of Education and the National Science Foundation. Strengthening 
science and math education is a core mission of the NSF. 

Science education is unique because it is concerned with the special character of 
science and its related disciplines — it is at once a body of knowledge and a dynamic 
questioning activity. Because of the nature of science it is important to have sci- 
entists involved in critical questions of science education. It was the recognition of 
this interdependence between scientists and the science education enterprise that 
drove the identification of science education as a key part of the NSF agenda when 
the agency was founded. 

We consider the NSF to be the engine of innovation for K-12 STEM education. 

The new NSF Education and Human Resources Division of Research on Learning 
in Formal and Informal Settings — known as DRL — is working to advance discovery 
and innovation at the frontiers of STEM learning and teaching. NSF supports the 
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highly innovative models and approaches to learning in formal and informal set- 
tings. NSF works to advance equity and participation for all, to foster linkages be- 
tween STEM education research and practice, and to unite education research and 
evaluation activities across the Foundation and with other federal agencies. 

The NSF has the capacity to incorporate the best from both the science and edu- 
cation R&D communities and can enlist scientists, academicians and researchers in 
a peer review process that generates and tests innovations in science-related dis- 
ciplines for education. Unlike the Department of Education, the NSF has the ability 
to tap into basic cognitive research, fold in new content and new ways of teaching 
this content from the disciplines, and explore new technologies for the delivery of 
professional development and for assessing teachers and their students. 

One of the most effective education programs at NSF is the Math and Science 
Partnerships. An analysis of 123 schools participating in the NSF MSP program 
shows continued increases in student proficiency in math and science since the pro- 
gram was first established in 2002. Students showed the most significant improve- 
ments in mathematics proficiency, with a 13.7 percent increase for elementary, 6.2 
percent increase for middle school, and 17.1 percent increase for high-school stu- 
dents. Science proficiency at each level showed marked gains as well, with a 5.3 per- 
cent increase for elementary, 4.5 percent increase for middle school, and 1.4 percent 
increase for high-school students. 

African-American, Hispanic, and white students showed significant improvements 
in elementary level mathematics, as did students designated as special education or 
as limited English-proficiency students. 

In addition to working with NSF on a MSP grant, NSTA has worked directly with 
federal agencies such as NASA, NOAA, and FDA to develop a combination of face- 
to-face training and online experiences that we believe has the potential to reach 
hundreds of thousands of K-12 science teachers. 

NASA, NOAA, DOT and the FDA have partnered with NSTA to develop SciPacks 
on topics supporting their mission. SciPacks are designed for educators who want 
or need to learn core science content. SciPacks contain three to five Science Objects, 
which are stand-alone, content-based units aligned with National Science Education 
Standards and Benchmarks for Science Literacy. These discrete online learning ex- 
periences are especially beneficial to teachers who are forced to teach out-of-field, 
elementary and middle level teachers who lack degrees in science, or those who 
need to increase their science knowledge of a particular content area. 

Each SciPak also contains a pedagogical implications section highlighting age-ap- 
propriate concepts and common student misconceptions. Teachers utilizing SciPaks 
get individualized e-mail support from a content expert and can complete a graded 
assessment demonstrating content mastery. 

SciPacks recently unveiled in the NSTA Learning Center focus on these content 
areas: Gravity and Orbits; the Universe and the Solar System; Earth, Sun and 
Moon; Coral Reef Ecosystems; Ocean’s Effect on Weather and Climate; Plate Tec- 
tonics, The Rock Cycle, Force and Motion and Energy. 

In addition, thousands of teachers have taken advantage of weekly NSTA Web 
Seminars on these topics. In addition NSTA Symposiums provide face-to-face train- 
ing with experts on these content areas from federal agencies, who interact one-on- 
one with K-12 teachers. 

Other STEM education programs from federal agencies are promoted extensively 
through NSTA print and online channels, and on the NSTA website. These commu- 
nication vehicles reach hundreds of thousands of teachers, teacher leaders, and oth- 
ers in the science education community. 

Federal agencies also share information about programs for science educators at 
the NSTA conferences, which draw approximately 25,000 teachers each year. 

During the last NSTA annual conference the National Institutes of Health fea- 
tured the NIH Research Zone, a coordinated effort that involved 27 institutes and 
centers from NIH professional societies and other supporting partners. The NIH Re- 
search Zone provided one stop shopping for teachers interested in discovering the 
resources available from the NIH research community. The groups represented in- 
cluded the National Biomedical Imaging and Bioengineering; the National Center 
for Research Resources; the National Human Genome Research Institute; the Na- 
tional Institute of Allergy and Infectious Diseases; the National Institute of General 
Medical Sciences; the National Institute of Neurological Disorders and Stroke; the 
National Institutes of Health Office of Science Education; the National Library of 
Medicine; and the Society for Neuroscience. 

In addition, the NIH Office of Science Education provides medical and life science 
curriculum supplements for grades K-12, as well as posters and videos promoting 
health science careers. 
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Workshops and exhibits on NASA’s education programs are also prominent at 
NSTA conferences. These include the NASA Educator Astronaut Launch, where 
teachers can join NASA’s first educator astronaut, Barbara Morgan, on her flight 
to the International Space Station later this summer. NASA is offering a website, 
classroom activities and challenges to teachers and students. 

Other NASA programs highlighted at the conference include the Student Observa- 
tion Network, 21st Century Explorer, and the Engineering Design Challenge. These 
programs allow students to use NASA data to conduct their own analyses and apply 
engineering principles to solve scientific problems. The NASA Smart Skies features 
a web-based simulator with real world air traffic control motion problems between 
two or more planes. Students apply proportional reasoning and distance rate time 
relationships to resolve conflicts by changing plane routes and speeds. 

While I cannot speak to the efficacy or the outcomes of these federal programs, 
we have found that many of these programs do provide key research and content 
to classroom teachers and help to excite teachers and students about science. One 
of the challenges with federal education programs, however, is that they reach only 
a minuscule proportion of our nation’s science teachers. We must find new ways to 
get proven, effective professional development programs up to scale so they reach 
a large number of teachers. 

Why is professional development so important? Last year the National Research 
Council report titled Taking Science to School: Learning and Teaching Science in 
Grades K-^8 said that professional development was key to supporting effective 
science instruction in the critical, early years of a child’s education. The NRC called 
for a dramatic departure from current professional development practice, both in 
scope and kind. 

All teachers need opportunities to deepen their knowledge of the science content. 
The NRC also believes that teachers need opportunities to learn how students learn 
science and how to teach it. They need to know how children’s understanding of core 
ideas in science builds across K-8, not just at a given grade or grade band. Teachers 
need to learn about the conceptual ideas that students have in the earliest grades 
and their ideas about science itself. They need to learn how to assess children’s de- 
veloping ideas over time and how to interpret and respond (instructionally) to the 
results of assessment. 

In short, teachers need opportunities to learn how to teach science as an inte- 
grated body of knowledge and practice — to teach for scientific proficiency. They need 
to learn how to teach science to diverse student populations, and to provide ade- 
quate opportunities for all students to learn science. 

We believe federal agencies have a key role in providing programs that will en- 
hance teacher content knowledge, help them to deliver effective instruction, and pro- 
vide insight into how students learn. 

It is interesting to note that the NRC report also asserts that “Federal agencies 
that support professional development should require that the programs they fund in- 
corporate models of instruction that combine the four strands of science proficiency; 
focus on core ideas in science; and enhance teachers’ science content knowledge, 
knowledge of how students learn science, and knowledge of how to teach science.” 

Looking to the future we anticipate that the soon-to-be released Academic Com- 
petitiveness Council report on the myriad of federal STEM education programs will 
bring about needed changes. 

From our observations, there is an overlap in many of the programs offered at 
the federal level. 

There is no oversight entity at the federal level that works to coordinate these 
STEM programs. The federal agencies do not appear to work together to facilitate 
the dissemination of research, or to discuss possible new ideas and avoid duplicative 
programs. 

We believe that better coordination and communication is desperately needed 
among federal agencies, bureaus, divisions, and centers that are involved with 
STEM education research and programs. 

Finally an inventory of STEM education programs across the federal agencies 
would inform future priorities and initiatives. Federal agencies should also work to 
coordinate their STEM education initiatives with states, local districts, the higher 
education community, and other key stakeholders. 

Improvements in STEM education require a commitment of leadership at the 
local, state, and federal levels. Education programs at the federal agencies will al- 
ways have a critical role to play in improving STEM education. We hope that any 
changes to existing programs, especially at the National Science Foundation, that 
may be come about as a result of the Academic Competitiveness Council report will 
be carefully reviewed and considered. 
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Thank you for allowing me the opportunity to address you today and I look for- 
ward to any questions you may have. 

Biography for Linda K. Froschauer 
National Science Teachers Association President, 2006-2007 

Linda K. Froschauer, K-8 Science Department Chair at the Weston Public 
Schools, in Weston Connecticut, is president of the National Science Teachers Asso- 
ciation (NSTA). She began her one-year term on June 1, 2006. 

Froschauer has been a devoted teacher and dedicated leader in science education. 
She began her teaching career as an elementary school teacher in Matteson, Illinois; 
moved on to middle level teaching at the Greenwich Public Schools, in Greenwich, 
Connecticut; and has been with the Weston Public Schools since 1985. She combines 
her work in the classroom with a leadership role in her school, serving as grades 
K-8 Science Department Chair/mentor teacher. Outside the classroom she has 
worked as an instructor for Chicago’s Museum of Science and Industry; as a writer/ 
consultant for many publications; and as a field editor, reviewer, and consultant for 
numerous organizations. 

For more than 30 years, Froschauer has been a leader and active member of 
NSTA. In 1976, she was named the first Preschool/Elementary Division Director to 
serve on the NSTA Board of Directors. She later worked on many NSTA commit- 
tees, including the International Convention Planning Committee, the Preschool/El- 
ementary Committee, and the Informal Education Committee, and she has chaired 
both the Awards and Recognition Committee and the Committee on Nominations. 
She also has served as Middle Level Division Director, worked on the Committee 
and Board Operations Task Force, and led the development of NSTA’s first Family 
Science Day, which was held in conjunction with the NSTA National Convention in 
Boston. 

Froschauer’s devotion to science education is evidenced by her involvement in nu- 
merous other professional organizations. She has served as president of the Con- 
necticut Science Supervisors Association (CSSA), the National Middle Level Science 
Teachers Association (NMLSTA), and the Council for Elementary Science Inter- 
national (CESI). She is also a member of the Connecticut Academy for Education 
in Mathematics, Science, and Technology; the Association of Presidential Awardees 
in Science Teaching; and the Society of Elementary Presidential Awardees. She has 
been actively involved in Project 2061, a national effort to improve science education 
sponsored by the American Association for the Advancement of Science. 

Froschauer was chosen as a Connecticut Science Educators Fellow and named 
Weston Teacher of the Year in 1999. Her other awards and accomplishments include 
receiving the NSTA Distinguished Teaching Award, Middle Level, in 2001; National 
Board for Professional Teaching Standards certification, also in 2001; the CSSA 
Charles Simone Award for Outstanding Leadership in Science Education in 1998; 
a Presidential Award for Excellence in Mathematics and Science Teaching in 1993; 
and the Educational Press Association of America’s Distinguished Achievement 
Award in 1991. 

Froschauer earned a BS degree in education from Northern Illinois University, an 
MA in science teaching from Governors State University, and a sixth-year degree 
in curriculum and supervision from Southern Connecticut State University. 

Chairman Baird. Thank you, Ms. Froschauer. Mr. Lach. 

STATEMENT OF MR. MICHAEL C. LACH, DIRECTOR OF 

MATHEMATICS AND SCIENCE, CHICAGO PUBLIC SCHOOLS 

Mr. Lach. Thank you, Mr. Chairman, Members of the Sub- 
committee. Thank you for inviting me here today to speak to you 
about this issue. It is an honor to sit before you along side col- 
leagues who I have worked with and learned much from. 

We have made great progress with mathematics and science in- 
struction in Chicago. Student performance has risen considerably 
over the past five years, and our rate of improvement is greater 
than that of the rest of the state. To do this, we developed a com- 
prehensive plan to coordinate all aspects of mathematics and 
science improvement which we call the Chicago Math and Science 
Initiative. As part of this work, we created a vision for high-quality 
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instruction; built the support infrastructure to provide high-quality, 
content -rich, professional development to thousands of teachers 
over the course of an academic year; forged partnerships with local 
businesses, museums, laboratories, and universities to increase the 
content knowledge of our teachers; and enhanced their after-school 
offerings to include mathematics and science enrichment. We have 
done this in the traditional urban context. Most of our students are 
poor, our facilities are crumbling, and we are limited on resources. 

I would argue that there are two major assets of the federal R&D 
mission agencies that will help K-12 STEM education. The first 
asset is human capital. The scientists and engineers of NASA, 
NOAA, NIST, EPA, and the Department of Energy are the best 
and brightest in the world. They are the ones making new discov- 
eries, creating new technologies, and literally exploring new worlds. 
The more we can connect students, parents, and teachers with 
their insights, energy, and perspectives, the better. 

The second major asset is the facilities. The laboratories and 
tools that are part of a federal R&D infrastructure are top notch, 
the particle accelerators, the spacecraft, the computers, the data 
sets. Most of our students have a very incomplete picture of the 
real work of scientists and engineers. Many teachers have never 
been part of a real scientific project. The facilities that are part of 
the federal R&D mission agencies should be utilized not only to 
ground science learning in a well-defined context but to enable stu- 
dents and teachers to grasp a vision of what they are trying to do. 

Communication between districts and the federal R&D mission 
agencies generally differs by the amount of collaboration that is in- 
tended in the partnership. The projects that are designed by the 
federal R&D mission agencies, individual teachers and schools find 
them by the usual methods, NSTA mailings and publications, 
websites, email lists. We regularly email our teachers of any oppor- 
tunities that we hear about, and generally because of our lack of 
resources, it is unconscionable for me not to encourage our teachers 
to participate in anything. 

The more strategic partnerships, programs are often developed 
jointly and are the result of an ongoing dialogue so the strengths 
of the partnering institutions are all leveraged. These partnerships 
require intense collaboration and flexibility from all sides as well 
as resources to support and create and maintain them. In my writ- 
ten comments, I have mentioned several partnerships we have 
used with Fermilab and Argonne National Laboratory. 

The federal R&D mission agencies have an important role to play 
in improving K-12 STEM education. By leveraging the human cap- 
ital and facilities these possess and connecting these to the existing 
plans and strategies of districts we will collectively be able to ad- 
vance the mathematic and science achievement of our students. 

Thank you. 

[The prepared statement of Mr. Lach follows:] 

Prepared Statement of Representative Michael C. Lach 

Mr. Chairman and Members of the Subcommittee, thank you for inviting me here 
today to speak to you about this issue. It is an honor to sit before you alongside 
colleagues whom I’ve worked with and learned much from. 
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I am the Director of Mathematics and Science for the Chicago Public Schools. The 
Chicago Public School system consists of over 600 schools, nearly 25,000 teachers, 
and more than 400,000 students. 

We have made great progress with mathematics and science instruction in Chi- 
cago. Student performance has risen considerably over the past five years, and the 
rate of improvement is faster than that of the state. (See Figure 1 and Figure 2.) 
To do this, we developed a comprehensive plan to coordinate all aspects of mathe- 
matics and science improvement, which we call the Chicago Math & Science Initia- 
tive. As part of this work, we created a vision for high quality instruction; built the 
support infrastructure to provide high quality, content-rich professional develop- 
ment to thousands of teachers over the course of an academic year; forged partner- 
ships with local businesses, museums, laboratories, and universities to increase the 
content knowledge of our teachers; and enhanced our after-school offerings to in- 
clude mathematics and science enrichment. 

We’ve done this in a challenging context. Eighty-five percent of our students come 
from low-income families. Our resources are low; Illinois ranks 47th in the Nation 
in the level of state support for education. Our capacity is limited — less than five 
percent of our K-8 teachers possess a State endorsement in mathematics. The Chi- 
cago Public Schools is an extremely decentralized school district. By State law, deci- 
sions about local school budgets, principal contracts, and curriculum are made by 
an elected body called the “Local School Council,” not the Chief Executive Officer. 

While we feel proud of our accomplishments, we know that we still have much 
work to do. An achievement gap remains in many of our schools. The number of 
students meeting and exceeding standards remains far too low. Our high schools, 
in particular, still have graduation rates that are not acceptable. 

In as much as possible, we connect with external resources to help us improve 
mathematics and science in the Chicago Public Schools. Much of the intellectual de- 
sign of our work comes from insights my colleagues on this panel have provided, 
from Dr. Nelson’s leadership, to Dr. Weiss’s insightful evaluations of large-scale 
change efforts, and to the National Science Teacher Association’s consistent support 
for teachers. Most of the funding for our efforts comes from the district; we work 
in every manner possible to leverage additional funding from corporate and univer- 
sity partners in the Chicagoland area. In particular, we’re happy to have several 
major universities that we work with in close partnership, and our relationship with 
Argonne National Laboratory has resulted in several programs that we have en- 
acted together. 

The gaps we face, and the resource and capacity limitations that we operate 
under make it unconscionable for us to turn down assistance. So my most important 
point today is that we really depend on the assistance and partnership of others — 
including the federal R&D mission agencies. They have an important role to place 
in science, technology, engineering, and mathematics education in the United 
States. 

I’d argue that there are two major assets of the federal R&D mission agencies 
that will help K-12 STEM education. As the Committee considers the most appro- 
priate way to connect these agencies with K-12 teachers and schools, programs 
should be designed so that these assets are highlighted. 

The first asset is human capital. The scientists and engineers of NASA, NOAA, 
NIST, EPA and DOE are the best and brightest in the world. They are the ones 
making new discoveries, creating new technologies, and literally exploring new 
worlds. The more we can connect students, parents, and teachers with their in- 
sights, energy, and perspectives, the better. The people of federal R&D mission 
agencies can both educate and inspire our students and teachers. A key priority 
should be to leverage this human capital so that they can assist schools and school 
districts in their work. 

The second major asset is the facilities. The laboratories and tools that are part 
of the federal R&D infrastructure are top notch — the particle accelerators, the space 
craft, the computers, the data sets. Most of our students have a very incomplete pic- 
ture of the real work of scientists and engineers. Many teachers have never been 
part of a real scientific project. The facilities that are part of the federal R&D mis- 
sion agencies should be utilized not only to ground science learning in a well defined 
context, but to enable students and teachers to understand a vision of what they’re 
trying to do. A second key priority of the federal R&D thus is to make the places 
where science and engineering are practices accessible in meaningful ways to stu- 
dents and teachers. 

I’d like to highlight a few examples of these that come from my experience with 
Argonne National Laboratory and Fermilab. 
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• The Academies Creating Teacher Scientists program provides summer intern- 
ships for teachers to conduct scientific research with ANL scientists . 1 In this 
program, both the human capital and the facilities of ANL are made available 
to select teachers in a sustained, supportive manner. 

• Fermilab’s Saturday Morning Physics sessions — in which I participated as a 
student — brings students to Fermilab to learn about modern physics topics 
and see real scientists in action . 2 Similarly, Argonne’s distance learning 
project uses modern technology to provide the opportunity for CPS students 
to meet and interact with professionals in technical fields . 3 Both of these pro- 
grams enable students to access the human capital and facilities of these lab- 
oratories. 

• The online ask-a-scientist provides a mechanism for student and teachers to 
get accurate answers to scientific questions from practicing scientists . 4 

• And, in an expression of our work as partners, the director of education at 
Argonne participates in our annual Principal For A Day project. 

Given these comments, a picture emerges about the sort of work that isn’t very 
helpful. Curriculum development is one. We know from decades of instructional ma- 
terial development that writing curriculum is a complicated, difficult process. More 
acutely, we know that robust curriculum is necessary but not sufficient for class- 
room improvement. In addition to strong materials, teachers need equipment, pro- 
fessional development workshops, coaching, and good assessments. Within a school, 
leaders need to understand how to support curriculum implementation, and manage 
improvement throughout grades and courses. Collections of lessons plans, by them- 
selves, are only a small piece of the puzzle. 

The proliferation of state and national standards and content also makes imple- 
mentation difficult. Special topics can be motivating and interesting to both teachers 
and students, but given the now-famous finding from the TIMSS study that our cur- 
riculum is “a mile wide and an inch deep,” adding more topics to cover only makes 
things difficult for teachers. If programs or projects are parochial, they’re harder to 
connect to our work. 

We also know that transforming classroom practice involves intensive capacity de- 
velopment sustained over time. It doesn’t happen over night — or in a one-day field 
trip or workshop. A brief visit to a laboratory or launch can be inspiring — and I 
don’t mean to downplay the importance of inspiring teachers and students about the 
world of science — but real change takes sustained work over time. Within an overall 
strategy, there’s certainly a need for both. 

I want to say a few words about the type of human capital development that we 
provide for teachers at the Chicago Public Schools. Our work falls into three major 
categories. The first is support for core instructional materials implementation, fo- 
cusing on the direct application of content and pedagogy to the classroom. This is 
almost always led by the district, and is difficult to conceive any outside institutions 
other than curriculum developers with the capacity to provide this work. The second 
is to enhance the content knowledge of teachers via university course work. The 
highly qualified teacher demands of the No Child Left Behind legislation as enacted 
in Illinois use course work as the main driver for this work. Seminars and sessions 
that don’t provide credit for teachers don’t enable me to very easily meet my goals. 
The third is activities that inspire the study of science and mathematics; generally, 
we use outside institutions such as museums and laboratories to do most of this 
work. 

In the Chicago Math & Science Initiative, the Chicago Public Schools was able 
to develop a coherent and comprehensive strategy for mathematics and science im- 
provement thanks to NSF systemic initiatives. It took us some time both to arrange 
the human capital and organization in order to structure such a strategy, but the 
results to date are quite positive. The more the federal R&D mission agencies can 
align their work to similar district strategies, the better the chance of success. With- 
out a clear connection to district’s vision, there will be no traction. A plan enables 
forward movement. And it takes resources to develop and drive such plans. 

When the proposals come to me as existing plans with little opportunity for local- 
ization, their chance of effectiveness is reduced considerably. Small programs that 
are aligned only peripherally to our strategies often just add complexity. We’ve had 
success because of our commitment to coherence, and the more the federal R&D 


1 http:/ / www.dep.anl.gov / p -k- 12 /acts / 

2 http: I / ed.fnal.gov /talks/ fermilab 1994 /web /ed-prog -sec -student. html#saturday 

3 http:/ / www.dep.anl.gov / p -k-12 / distancelearning / 

4 http:/ / www.newton.dep.anl.gov / 
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mission agencies can align with that, the better. I can’t think of any proposals that 
have come to me with an evaluation report documenting their effectiveness. 

Communication between districts and the federal R&D mission agencies generally 
differs by the amount of collaboration that is intended in the partnership. For 
projects that are designed by the federal R&D mission agencies, individual teachers 
and schools find them by the usual methods — NSTA mailings, websites, e-mail lists. 
We regularly e-mail our teachers any opportunities that we hear about. For more 
strategic partnerships, programs are often developed jointly and are the result of 
an ongoing dialogue so that the strengths of the partnering institutions are lever- 
aged. These partnerships require intense collaboration and flexibility from all sides. 

The federal R&D mission agencies have an important role to play in improving 
K-12 STEM education. By leveraging the human capital land facilities that these 
institutions possess, and connecting these to the existing plans and strategies of the 
district, we’ll collectively be able to advance the mathematics and science achieve- 
ment of our students. 

Answers 

1. How do you find resources for improving science and mathematics edu- 
cation in the Chicago Public Schools? 

Individual teachers find text, lesson plans, and other classroom resources via the 
usual methods — NSTA mailings, websites, e-mails. When the central office learns of 
opportunities such as this, we distribute them via e-mail to our schools. 

More strategic partnerships — such as the ones described above — come about via 
ongoing dialogue with our partners in museums, laboratories, and universities. 
These are generally designed together. 

2. What resources have you garnered from the federal R&D mission agen- 
cies? How has this contributed to improving your students’ under- 
standing of science? 

As mentioned, our partnerships in particular with Argonne National Laboratory 
and the Fermi National Accelerator Laboratory have resulted in several successful 
programs that connect teachers and students with the scientists, engineers, and fa- 
cilities of these institutions. 

3. What type of support that the federal R&D mission agencies could pro- 
vide would have the most impact on STEM education for your teachers 
and students in Chicago Public Schools? 

The most productive supports are those that (1) inspire students and teachers to 
study science and mathematics and (2) provide students and teachers with a deep 
understanding of the real-world work of scientists and engineers. Supports that are 
not particularly effective include (1) lesson plans and curriculum development, (2) 
workshops that don’t connect directly to specific instructional materials or univer- 
sity credit. To enable deeper collaboration, resources need to be allocated with the 
expressed purpose of connecting K-12 schools and districts with the federal R&D 
mission agencies. 
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Biography for Michael C. Lach 

Michael C. Lach is Director of Mathematics and Science for the Chicago Public 
Schools, overseeing mathematics and science teaching and learning in the 500 ele- 
mentary schools that comprise the Nation’s third largest school district. Mr. Lach 
began teaching high school biology and general science at Alcee Fortier Senior High 
School in New Orleans in 1990 as a charter member of Teach For America, the na- 
tional teacher corps. After three years in Louisiana, he joined the national office of 
Teach For America as Director of Program Design, developing a portfolio based al- 
ternative-certification system that was adopted by several states. Returning to the 
science classroom in 1994 in New York City Public Schools, and then back to Chi- 
cago in 1995 to Lake View High School, he was named one of Radio Shack’s Top 
100 Technology Teachers, earned National Board Certification, and was named Illi- 
nois Physics Teacher of the Year. He has served as an Albert Einstein Distinguished 
Educator Fellow, advising Congressman Vernon Ehlers (R-MI) on science, tech- 
nology and education issues. He was lead curriculum developer for the Investiga- 
tions in Environmental Science curriculum developed at the Center for Learning 
Technologies in Urban Schools at Northwestern University and published by It’s 
About Time, Inc. He has written extensively about science teaching and learning for 
publications such as The Science Teacher, The American Biology Teacher, and Sci- 
entific American. He earned a Bachelor’s degree in physics from Carleton College, 
and Master’s degrees from Columbia University and Northeastern Illinois Univer- 
sity. 

Chairman Baird. Thank you, Mr. Lach. Dr. Nelson. 
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STATEMENT OF DR. GEORGE D. NELSON, DIRECTOR OF 

SCIENCE, TECHNOLOGY, AND MATHEMATICS EDUCATION, 

WESTERN WASHINGTON UNIVERSITY 

Dr. Nelson. Good morning, Mr. Chairman, and Members of the 
Committee. My name is Pinky Nelson, and today I am wearing my 
science educator hat. 

What resources can the mission agencies focus on? The two goals 
of literacy and workforce development, have skilled and knowledge- 
able workforce of scientists, engineers and technicians engaged in 
cutting-edge science and technology development focused on mis- 
sions critical to the country, research and technology partnerships 
with industry and universities, world-class and unique laboratories 
and facilities, and the capacity for long-term funding. What re- 
sources do the mission agencies generally lack? Knowledge of the 
K-12 education system and how it is structured and regulated, in- 
ternal expertise in education research, curriculum development, ef- 
fective instruction, or teacher preparation. The agencies should 
combine their programs so those that can take advantage of their 
strengths and be sure to include appropriate partners when work- 
ing in areas where they lack the expertise. They have the capacity 
to sustain and grow programs that are working and axe those that 
are not. 

Possible areas where I think the mission agencies can contribute 
effectively include career pathways for high-school students and 
mission-related undergraduate and graduate research. More chal- 
lenging is participation in K-12 curriculum development and eval- 
uation, and teacher preparation. 

I will briefly discuss the areas of challenge before moving on to 
discuss career pathways and support for research. Working to- 
wards achieving universal literacy by improving K-12 schooling re- 
quires deep collaboration with professionals across the education 
system, often in a non-leadership role, creating the capacity and 
improving the system comes first, the agencies’ short and inter- 
mediate-term goals come second. 

There is a huge inventory of poorly-designed and under-evalu- 
ated mission-related curricula, posters, and lesson plans and asso- 
ciated professional development rarely used in classrooms and with 
no natural home in a coherent standards-based curriculum. The 
constant barrage of new resources adds to the noise in the system 
and contributes to the mile-wide, inch-deep problem. However, I do 
have one positive example. I recently received a copy of an astron- 
omy curriculum for grades three to five that was developed collabo- 
ratively by NASA and the Professional Science Educators and De- 
velopers at the Lawrence Hall of Science at UC-Berkley. It is high- 
quality and fills a real need for instructional materials at this level. 
A collaborative curriculum development model such as this is rare. 
Adding the evaluation component could make it exemplary. 

My current work includes exploring the preparation of effective 
new STEM teachers and helping current teachers improve their 
practice. This is not a part-time job or one for the faint of heart. 
Agencies should encourage and provide incentives for their STEM 
retirees to become teachers. In addition, they should collaborate 
with excellent teacher preparation programs and support their rig- 
orous evaluation. 
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In high schools and community colleges, agencies can collaborate 
with appropriate education organizations and industry to develop 
and support career pathways for students, for example, in high 
school in high-need areas like photonics or nanotechnology. The 
agency can promote its mission through carefully designed, imple- 
mented, and evaluated technology programs targeting the future 
workforce. These programs can take full advantage of the agency 
talent pool. The NSF Advanced Technology Education Program has 
created some effective models at the community college level. Agen- 
cies could expand this work, bring it into high school, career, and 
technical education programs and provide sustaining funding that 
is not available from NSF R&D programs. 

Research scientists, engineers, and technicians can help muse- 
ums or other informal education entities display and communicate, 
both in real and cyberspace, the new science and technology that 
is coming out of the agencies to excite and inform students, par- 
ents, and voters. Additionally, the personal stories of STEM work- 
ers at all levels, including clear maps of the paths through school 
that qualify them for those jobs can help motivate students to enter 
career pathways. 

Agencies can support undergraduate, graduate, and postdoctoral 
students to engage in mission-related research and then hire the 
best of them into meaningful jobs. They can provide undergraduate 
and graduate students authentic research experiences in their cen- 
ters and laboratories, again with the prospect of meaningful jobs. 
As a graduate student, I spent two invaluable summers at the Air 
Force Cambridge Research Laboratory solar observatory in Sun- 
spot, New Mexico. 

The NASA Space Grant program in Washington State is a posi- 
tive example. NASA funds leveraged with a one-to-one match sup- 
port around 150 graduate students every year to engage in STEM 
research mentored by faculty at institutions throughout the state, 
internships at companies, or NASA centers, or participation on stu- 
dent design teams. Last year 100 percent of the Space Grant schol- 
ar graduates went on to STEM graduate work or employment. 
While the program keeps good statistics, it could benefit from a 
more sophisticated effort. 

Thank you. I look forward to your questions. 

[The prepared statement of Dr. Nelson follows:] 

Prepared Statement of George D. Nelson 

Chairman Baird and Members of the Committee, it is a privilege to accept your 
invitation to participate in the hearing and provide my perspective on the STEM 
education programs of the federal mission agencies. 

My primary perspective comes from my recent roles in STEM education reform 
as Directory of Science, Mathematics, and Technology Education at Western Wash- 
ington University, and my previous position as Director of Project 2061 at the Amer- 
ican Association for the Advancement of Science. I am also Principal Investigator 
of a targeted Mathematics and Science Partnership grant from NSF that brings to- 
gether 28 regional school districts, Washington State LASER, three state commu- 
nity colleges, the Northwest Indian College, and Western Washington University in 
an effort to reform science education with a particular focus on improving K-16 
science teacher preparation. 

Personal experiences from previous positions have profoundly influenced my per- 
spective towards STEM education and general education reform. I have worked as 
a research astrophysicist, flown three missions on the U.S. Space Shuttle as a 
NASA astronaut, served as Associate Vice Provost for Research at the University 
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of Washington, and taught at all levels in higher education. I have spent consider- 
able time thinking about and engaging in discussions with NASA and the Depart- 
ment of Energy about their K-12 education programs, and served on numerous ad- 
visory committees, commissions, and on boards of directors including the Pacific 
Science Center, the Art Institute of Seattle, and the Center for Occupational Re- 
search and Development (CORD). I am also the proud father of a dedicated and out- 
spoken middle school mathematics and science teacher from Katy, Texas. 

This testimony will focus on the role of the federal mission agencies, but it is al- 
ways good to keep the big picture in mind. The American education system is enor- 
mous, with over 50 million students and 3.1 million teachers. Counting the critical 
role of STEM learning in the elementary grades, more than half of these teachers 
are responsible for teaching mathematics and science. The system is also decentral- 
ized, locally funded and governed, and subject to myriad regulations. Mr. Lach has 
provided a compelling picture of the Chicago system. There are 15,000 other dis- 
tricts in America, each with its own unique strengths and challenges. 

Since the federal mission agencies depend so heavily on both a literate citizenry 
for continued support and STEM professionals at all levels to carry out their mis- 
sions, it is in the interest of the agencies to contribute appropriately to achieving 
two STEM education goals: 1) universal math and science literacy and 2) signifi- 
cantly increasing the number and diversity of American students entering and suc- 
cessfully exiting the STEM pipeline. 

I shall now address the Committee’s specific questions. To approach a model for 
how the federal mission agencies can contribute, it is reasonable to ask, what re- 
sources can the mission agencies focus on the two goals of literacy and workforce 
development? Here is my short list. 

• A skilled and knowledgeable workforce of scientists, engineers, and techni- 
cians engaged in cutting edge science and technology development focused on 
missions critical to the country 

• Research and technology partnerships with industry and universities 

• World-class and unique laboratories and facilities 

• Long-term funding. 

It is also important to ask, what resources do the mission agencies generally lack? 

• Knowledge of the K-12 education system, how it is structured and regulated 

• Internal expertise in education research, curriculum development, effective 
instruction, or teacher preparation. 

1. In what ways can federal R&D mission agencies contribute most effectively to im- 
prove K-12 STEM education ? Can you give examples of particularly effective pro- 
grams? 

Taking advantage of their strengths, agency professionals can collaborate with ap- 
propriate education organizations and industry to develop and support Career Path- 
ways for students in high schools and community colleges, for example in high need 
areas like photonics or nanotechnology. The agency can promote its mission through 
carefully designed, implemented, and evaluated technology education programs tar- 
geting the future workforce. These programs can take full advantage of the agency 
talent pool. The NSF Advanced Technology Education program has created some ef- 
fective models at the community college level. Agencies could expand this work, help 
bring it into high school Career and Technical Education programs, and provide sus- 
taining funding that is not available from NSF R&D programs. 

Research scientists, engineers, and technicians can help museums or other infor- 
mal education entities display and communicate — both in real- and cyberspace — the 
new science and technology that is coming out of the agencies to excite and inform 
students, parents, and voters. Additionally, the personal stories of STEM workers 
at all levels, including clear maps of the paths through school that qualify them for 
those jobs can help motivate students to enter the Career Pathways. 

My current work includes exploring the preparation of effective new STEM teach- 
ers and helping current teachers improve their practice. This is not a part-time job, 
or one for the feint of heart. Agencies should encourage and provide incentives for 
their STEM retirees to become teachers, again making use of their talented work- 
force. They should also collaborate with excellent teacher preparation programs and 
support their rigorous evaluation. Poor preparation for entering the classroom re- 
sults in ineffective instruction and low retention. 

2. At the undergraduate level, what type of support could the federal R&D mission 
agencies provide that would recruit more students into pursuing careers in the 
physical sciences? 
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Agencies can support undergraduate, graduate, and postdoctoral students to en- 
gage in mission-related research, and then hire the best of them into meaningful 
jobs. They can support students on campuses to work with faculty engaged in mis- 
sion-relevant research. They can also provide undergraduate and graduate students 
authentic research experiences in their centers and laboratories — again with the 
prospect of meaningful jobs. As a graduate student, I spent two invaluable stints 
at the Air Force Cambridge Research Laboratory solar observatory in Sunspot, New 
Mexico engaged in cutting edge research with world-class instruments. 

The NASA Space Grant program in Washington State is a positive example. 
NASA funds support around 150 undergraduate students every year to engage in 
STEM research, mentored by faculty at institutions throughout the state, intern- 
ships at companies or NASA centers, or participation on student design teams. Last 
year 100 percent of the Space Grant scholar graduates went on to STEM graduate 
work or employment. While the program keeps good statistics, it could benefit from 
a more sophisticated evaluation effort. 

3. How does the lack of coordination and overarching strategy for STEM education 

programs hinder the agencies from making an impact ? 

There is a huge inventory of poorly designed and under-evaluated mission-related 
curricula (posters and lesson plans and associated professional development) rarely 
used in classrooms and with no natural home in a coherent standards-based cur- 
riculum. The constant barrage of new “resources” adds to the noise in the system 
and contributes to the “mile wide, inch deep” problem. Effective curriculum develop- 
ment requires a deep collaboration with a team of professional curriculum devel- 
opers, education researchers, and classroom teachers. 

In that light, I do have one positive example. I recently received a copy an astron- 
omy curriculum for grades 3-5 that was developed collaboratively by NASA and the 
professional science educators and developers at the Lawrence Hall of Science and 
UC-Berkeley. It is high quality and it fills a real need for instructional materials 
at this level. A collaborative curriculum development model such as this is rare. 
Adding a rigorous evaluation component to explore ho well the curriculum helps 
teachers teach and students learn could make it exemplary. 

Summary 

A focus on 1 ) partnering with high schools and community colleges along with ap- 
propriate education professionals and industry partners on mission-related tech- 
nology education programs for the future technical workforce, and 2) supporting 
mission-related research for undergraduate and graduate students both in agency 
facilities and on university campuses could pay major dividends. This would require 
an achievable overarching strategy, but not necessarily significant coordination 
among the agencies. The critical collaboration would be with STEM education pro- 
fessionals (not just K-12 teachers), university faculty, and industry partners. 

Biography for George D. Nelson 

Dr. George D. Nelson is the Director of Science Mathematics, and Technology 
Education and Professor of Physics and Astronomy at Western Washington Univer- 
sity in Bellingham, Washington. The program is responsible for the preparation of 
future K-12 science, mathematics, and technology teachers. It is also a research and 
development center with a focus on teacher preparation and science, mathematics, 
and technology education reform. He is currently the principal investigator on a $12 
million NSF project, the North Cascades and Olympic Science Partnership. 

Prior to joining Western Washington University in 2002, Dr. Nelson was Director 
of Project 2061 and a member of the senior staff of the American Association for 
the Advancement of Science. Project 2061 is engaged in the reform of science, math- 
ematics, and technology education at all levels with a focus on helping to create a 
system where all high school graduates are literate in science, mathematics, and 
technology. Under Dr. Nelson’s leadership Project 2061 produced a number of 
ground-breaking publications including Blueprints for Reform, Designs for Science 
Literacy, and the Atlas of Science Literacy. The project also developed unique and 
rigorous procedures for evaluating curriculum materials and assessments, and 
greatly expanded its professional development activities. 

From 1989 to 1996 Dr. Nelson was Associate Vice Provost for Research and Asso- 
ciate Professor of Astronomy and Education at the University of Washington. His 
administrative responsibilities included research policy, government-university-in- 
dustry interactions, university-K-12 education interactions, and federal relations. 
He taught graduate courses in stellar atmospheres and solar physics and under- 
graduate courses in general astronomy. In the college of education he taught an in- 
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novative seminar on science education for scientists, graduate students, and teach- 
ers using Project 2061 as the underlying foundation. During the 1992-93 academic 
year, Dr. Nelson was a fellow of the American Council on Education. 

From 1978 to 1989 he served as a NASA astronaut and flew as a mission spe- 
cialist aboard three Space Shuttle flights. These missions included the first on-orbit 
satellite repair in 1984. Dr. Nelson was the pilot of the first operational flight of 
the manned maneuvering unit and the primary extra-vehicular crewman. He also 
served on the crew of the flight of Discovery in September 1989 immediately fol- 
lowing the loss of the Challenger and was extensively involved in the rework of all 
crew procedures and the re-engineering of Space Shuttle components and software. 
He has advised NASA through service on a number of committees, most recently 
as Chair the Hubble Space Telescope Servicing Missions 3A and 3B External Inde- 
pendent Readiness Review Team. 

Dr. Nelson has served on several boards of directors including the Art Institute 
of Seattle, Analytic Services Inc., and the Pacific Science Center. He received his 
B.S. in physics from Harvey Mudd College and M.S. and Ph.D. in astronomy from 
the University of Washington. His research interests include science education, edu- 
cation reform, and radiative transfer and hydrodynamics applied to interesting prob- 
lems in astrophysics. 

He lives in Bellingham with his wife, Susie. They have two grown daughters; 
Aimee Nelson-Engle and Marti Nelson-Frazier and three perfect grandsons, Pierce, 
Langston, and Andrew. 

Chairman Baird. Thank you, Dr. Nelson. Mr. Van Reiner. 

STATEMENT OF MR. VAN R. REINER, PRESIDENT AND CEO, 

MARYLAND SCIENCE CENTER, MARYLAND ACADEMY OF 

SCIENCES 

Mr. Reiner. Good morning, Mr. Chairman and Members of the 
Subcommittee. I am Van Reiner, President of the Maryland Science 
Center located in Baltimore’s Inner Harbor. We have three levels 
of interactive, hands-on exhibits, a planetarium, an IMAX theater, 
classroom space, and a program space for live science demonstra- 
tions. Each year we admit 100,000 students attending with school 
groups to augment their science and mathematics curriculum. We 
are a member of the Association of Science Technology Centers, In- 
corporated. 

We, like hundreds of institutions across the country, employ 
what is known as informal education as a way to connect people 
with science and technology. Learning by doing is the basis for our 
approach. Showing how, rather than stating why, gives visitors the 
information they need to make informed decisions about how to re- 
late the topic at hand. When we are successful, we go from global 
to local to individual by giving the facts, a better understanding of 
the topic or how it relates to them, and hopefully a quest for more 
knowledge. 

Motivating students to take interest in science, technology, engi- 
neering, and math, whether or not they choose to pursue a career 
in those fields, puts science centers in a unique position to spark 
an initial interest. 

Collaborations are essential to the success of science centers, and 
we have a history of collaborating with the federal R&D mission 
agencies. The longest running collaboration is with NASA. We have 
co-hosted events with NASA such as having students participate in 
televideo conferences with the astronauts aboard the International 
Space Station and the Shuttle, as well as watching a European 
solar eclipse while talking to NASA Goddard scientists on board a 
ship in the Black Sea. We have helped to develop after-school as- 
tronomy programming and are embarking on a citizen science 
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project to measure the amount of UV radiation that reaches the 
earth at Baltimore’s Inner Harbor. 

Scientists from federal agencies participate in our “scientist of 
the month” program to interact directly with our visitors to discuss 
current research findings. Other instances with NASA and other 
agencies are listed in my written testimony. 

Evaluation of these programs and exhibits have been performed 
in our institution by us. The accepted practice for informal edu- 
cation institutions such as ASTC members has been to do front- 
end, formative, remedial, and summative evaluations of the pro- 
gram or exhibit by a third party to be sure that stated goals of the 
project are met. These evaluations are required for NSF or NIH 
grants, and we use them for other federally sponsored exhibits and 
programs. Several of these evaluations are included in the attach- 
ments to my written testimony, and I apologize for the length of 
them. [Evaluations included in Appendix 2: Additional Material for 
the Record.] 

These evaluations are thorough and complete and help us to 
know if we have met the requirements of the project and if the au- 
dience understands the subject presented. We feel that without this 
evaluation we would quickly lose our relevancy. 

Currently we are collaborating with NASA and NOAA on evalu- 
ating a project called Science on a Sphere, the globe identical to the 
one that Queen Elizabeth II visited at Goddard Space Center last 
week. We have been asked by NOAA to lead the users group to 
work with the agencies to develop evaluation methods specifically 
for the exhibit as well as the programming that the group develops 
around the exhibit. This collaboration between NOAA and NASA 
is unique and should be encouraged. Scientists from the two agen- 
cies are working together to ensure that the data presented is clear 
and meaningful. Increased collaboration between federal R&D mis- 
sion agencies and science centers can better accomplish the goals 
of STEM education programs. The dialogue between federal mis- 
sion agencies should be expanded so that the general public and 
students can be presented with knowledge in larger and more 
meaningful ways. 

We believe that greater understanding leads to greater accept- 
ance that science is resident in everything we do. It just doesn’t 
happen in a laboratory. Science centers are a resource in every 
sense of the word and should be viewed as such. We know how visi- 
tors react and how to best present scientific discovery and scientific 
progress to the public. We believe greater utilization of science cen- 
ters as resources for federal R&D mission agencies is the best way 
to help raise the level of scientific literacy with the general public, 
including students. We can, and do augment, the formal classroom 
instruction using resources that would be either too expensive or 
too impractical for the classroom. I believe that federal R&D mis- 
sion agencies should be required to allocate a portion of their re- 
sources to educate the public as is now required by the National 
Science Foundation. 

I thank you for this opportunity, and I look forward to your ques- 
tions. 

[The prepared statement of Mr. Reiner follows:] 
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Prepared Statement of Van R. Reiner 
Chairman Baird, Ranking Member Ehlers, and Members of the Subcommittee: 

The Maryland Science Center, located at Baltimore’s Inner Harbor, is a private 
non-profit that had its genesis in 1797 as the Maryland Academy of Sciences, which 
is still our parent organization. The current building was put in service in 1976 and 
expanded in 2004. The facility houses three levels of highly interactive, hands-on 
exhibits, a planetarium, an IMAX Theater, classroom space, and program space for 
live demonstrations on a variety of scientific phenomena. 

Each year, Maryland Science Center welcomes over 400,000 visitors to our facility 
with about 100,000 students visiting with school groups to augment their science 
and mathematics curriculum. Major areas of concentration — our core programming 
areas — are Earth system science, space and aerospace science, health sciences and 
the human body, and early childhood education. All of the permanent exhibits at 
the Maryland Science Center were designed to be in concert with the Maryland De- 
partment of Education Science Curriculum, and where applicable, under the influ- 
ence of The Benchmarks for Science Literacy published by the American Academy 
for the Advancement of Science as part of Project 2061, the National Science Edu- 
cation Standards prepared under the auspices of the National Research Council, and 
the Principles and Standards for School Mathematics from the National Council of 
Teachers of Mathematics. 

The Maryland Science Center, like hundreds of institutions across the country 
and around the world, employ what is known in the field as “informal education” 
as its means of connecting people with science and technology. Learning by doing 
is the foundation of informal education, along with demonstrating practical exam- 
ples. Building on that foundation, the role of the science center as distiller and in- 
terpreter of the latest in scientific discovery and connecting the public, especially 
school children, to the everyday application of these advancements, is a key strategy 
we employ. Showing how, rather than stating why, gives visitors the information 
they need to make informed decisions about how they can relate to the topic at 
hand. When we’re successful, we go from global to local to individual along a con- 
tinuum giving the individual the facts, a better understanding or how it relates to 
them, and hopefully, a quest for more knowledge. Science centers in particular have 
an ability to affect change by engaging school children through their use of informal 
education methods. Motivating students to take interest in science, technology, engi- 
neering and math, whether or not they choose to pursue a career in those fields, 
puts science centers in a unique position to spark an initial interest. 

Collaboration has always been a means to an end for the institution. Seeking 
partnerships to ensure that our exhibits and programs are the best they can be has 
yielded quality educational and enjoyable product for our visitors. The Maryland 
Science Center has long cultivated a history of working with the National Aero- 
nautics and Space Administration (NASA), National Oceanographic and Atmos- 
pheric Administration (NOAA), National Institute of Standards and Technology 
(NIST), as well as the National Science Foundation (NSF) and the National Insti- 
tutes of Health (NIH). Working with these agencies, and others, we develop perma- 
nent exhibits, traveling exhibits, and programs designed to inform and educate the 
general public — from those school children to their parents and teachers — about not 
only the basics of science, math and technology, but also the latest events and dis- 
coveries in our core programming areas. 

In our core exhibits we use high tech, multimedia update centers known as Links. 
Each Link is designed to offer real time investigation of science topics in the news 
that are appealing to mass audiences. When something happens in the world of 
science and technology, our Link areas and Link staff key in on the news releases, 
scientific data, and information resources to process and present the findings in a 
relevant, thought-provoking manner. Our Links are designed to give visitors who 
seek cursory, as well as deeper understanding of science as it happens, a chance 
to speak with our content experts, and to explore areas that can be a resource for 
the quest for deeper knowledge. Maryland Science Center currently features three 
Links. TerraLink focuses on Earth system science, SpaceLink concentrates on space 
and aerospace science and BodyLink examines health sciences. The fabrication and 
design of these centers was accomplished with input from the aforementioned agen- 
cies, and we continue to interface with these agencies to stay on top of current re- 
search. 

In the SpaceLink update center, NASA has been a primary partner. The partner- 
ship has taken many forms. With the Goddard Space Flight Center, we have helped 
develop an after school astronomy club format. The results can be seen at 
www.afterschoolastronomy.org. This site is a resource that provides information for 
students with an interest in astronomy and put the information to use in practical 
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applications, a hallmark of informal education. We have also co-hosted individual 
events with Goddard such as Sun-Earth Day, where educators from Goddard are at 
the Maryland Science Center to explain that we live in the atmosphere of a star, 
which has many effects on our planet. We are part of a larger system and under- 
standing that system is vital to other scientific endeavors we attempt to explain. 
This is an annual event where visitors engage in activities and receive print mate- 
rials to take with them that allow them to further explore the topics at home. 
Again, relevancy and real world examples — informal education in process. 

Maryland Science Center and Goddard also co-hosted an Earth Explorer Institute 
where we convened 25 informal educators representing science centers and muse- 
ums from across America to discuss and recommend NASA Earth science program- 
ming for informal audiences. As an outgrowth of that, we are participating in a UV 
Citizen Science project. We will enlist citizens to measure the amount of UV radi- 
ation that hits the Earth at Baltimore’s Inner Harbor, enter the data into a nation- 
wide database, and then participate in ongoing work to measure changes in UV ra- 
diation across the country. People are exposed to scientific equipment, scientific 
methods, and will be able to participate in a nationwide study. They also gain a 
greater understanding of the implications of changing UV levels as it relates to their 
everyday lives. 

Maryland Science Center makes its resources available to provide opportunities 
for students to witness events such as a solar eclipse. In August of 1999, a group 
of students observed a solar eclipse in Europe via a link with Goddard scientists 
on board a research vessel in the Black Sea. Students came to the Maryland Science 
Center to learn about the eclipse and participated in two televideo conferences be- 
fore and during the event to observe and ask questions. 

In partnership with the Johnson Space Center, Maryland Science Center has 
hosted Baltimore City School students for four live downlinks from the International 
Space Station and the Space Shuttle since 2001. Students have conversations as 
well as question and answer sessions with the astronauts aboard the Station and 
the Shuttle. Prior to the downlinks, students visited the Maryland Science Center 
to learn about the specific mission and prepare questions for the astronauts. The 
missions included the delivery of the Destiny component to the International Space 
Station, and the Hubble Space Telescope servicing mission. After these two mis- 
sions, the entire Shuttle astronaut crew involved in the downlink visited the Mary- 
land Science Center to meet with the students who participated in the conference 
and the general public. We were the first informal education institution to have this 
opportunity, and mentored other museums on how to replicate the experience. 

With the Space Telescope Science Institute, we were advisors on the making of 
a short IMAX film entitled “Hubble: Galaxies Across Space and Time.” This three- 
minute film has been shown in over a dozen IMAX theaters nationwide. At the 
Maryland Science Center, it has had 1,564 screenings for over 110,000 visitors. We 
have also developed a planetarium show to explain what the Hubble has allowed 
us to see and how those images have helped to shape the way we view the universe 
as well as increasing our knowledge of our place in the universe. 

Through collaboration, the Maryland Science Center participates in other smaller 
programs funded by NASA as part of a larger grant to another institution. We will 
create a series of programs and events highlighting the New Horizons mission to 
Pluto and the Kuiper Belt. We will host a small exhibit, offer a teacher workshop 
and host a Pluto Family Science Night. Keying on the recent news and popular cul- 
ture references to Pluto’s status as a planet or not, Maryland Science Center will 
present the latest Pluto information as part of a popular planetarium show “Planet 
Trek.” In conjunction with the Howard Owens Science Center in the Prince George’s 
County, Maryland school system, we are to develop a planetarium program on Pluto 
and the New Horizons mission for distribution to school planetariums nationwide — 
currently numbering in excess of 600. 

In TerraLink, the Earth systems science update center, Maryland Science Center 
has partnered with NASA and NOAA to provide ongoing support for programming 
including visual material and scientific expertise. NASA and NOAA scientists peri- 
odically visit to work with students as part of our Scientist of the Month program 
and on special programming days such as Earth Day. Students and visitors have 
a chance to see science pursuits as both a vocational option and simply as a means 
to broaden awareness that science, technology, engineering and math is not a nar- 
row cast field of inquiry and exploration. The focus of this program is to provide 
science and technology careerists as role models for students as well as being able 
to offer another thread for visitors and students to seek out information about how 
things like atmospheric phenomena occurs. TerraLink staff and the Science Person 
of the Month collaborate to present topics and information to the public in under- 
standable terms. 
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Using NASA and NOAA data and visual imagery, as well as utilizing experts 
from the agencies in the Science Person of the Month program currently defines the 
extent of Maryland Science Center collaboration with the agencies named in this in- 
quiry. It should be noted however that prior to the institution’s recent expansion 
and broadening of its core competencies, the programmatic synergies between Mary- 
land Science Center and these agencies was limited by definition of scope and mis- 
sion. 

Currently BodyLink, the Maryland Science Center’s health sciences update center, 
collaborates and partners with other federal agencies — primarily NIH, through its 
Science Education Partnership Award program — but does not at this time enjoy a 
relationship with the agencies named in this inquiry. However, topics like the stud- 
ies of the effects on the human body of extended durations of time spent living in 
space are of interest to BodyLink staffers and Maryland Science Center and the op- 
portunities to collaborate and deliver programming similar to that which is already 
in place in SpaceLink and TerraLink are currently tracking with our institutional 
collaborative goals. 

Beyond the Link areas of our core exhibits and programming, and in partnership 
with NOAA, Maryland Science Center has embarked on an exciting project entitled 
Science On a Sphere (SOS). This is an Earth visualization system developed by 
NOAA that projects a wide variety of data sets onto a large sphere to create dy- 
namic global views of the entire Earth. Visitors observe hurricane development and 
prediction, tectonic plate movement and earthquakes, sea surface temperatures and 
their effect on global weather conditions, as well as observe global warming models 
and the potential effect on the Earth. Science On a Sphere is now a permanent ex- 
hibit at the Maryland Science Center. Having this technology also allows us to com- 
pare Earth to other planets and NASA data sets have been converted to show the 
Moon, Mars, Saturn and the Sun on the same sphere. We have, in collaboration 
with NOAA and NASA, developed Maryland Science Center staff-delivered, visitor- 
centered, programs as well as produced prerecorded programs that explain the im- 
ages being observed. 

We have also developed traveling exhibits. NIST was instrumental in providing 
technical information for our Titanic Science exhibit. NIST performed analyses of 
rivets from the Titanic hull that were found at the wreck site to determine the 
strength level of the rivets, so our information would be factual. One of the ques- 
tions surrounding the Titanic disaster was whether or not the steel used to make 
the rivets was of poor quality — and if that might explain how the “watertight” fea- 
tures failed. By presenting the data and the surrounding conditions, visitors were 
left to their own conclusions as to how the rivets might have contributed to the 
Titanic’s end. 

Although we were not asked to speak directly to our collaboration with other fed- 
eral agencies, two examples of Maryland Science Center collaboration with the Na- 
tional Institutes of Health are funding and content expertise for a traveling exhibit 
titled: The Changing Faces of Women’s Health and funding and content expertise 
for our permanent health sciences update center BodyLink. 

With the exception of Science On a Sphere, the evaluation of permanent exhibits 
has been done by the Maryland Science Center. The accepted practice for informal 
education institutions, as exemplified by the Association of Science-Technology Cen- 
ters, (ASTC), has been to include a front-end evaluation as part of any project. This 
involves determining what the public knows about the subject through focus groups 
and questionnaires. The project is then judged as to what is feasible to build and 
install, and through prototyping of exhibit pieces, determining if the public will un- 
derstand the idea or concept presented by the various exhibit pieces. When the 
project is complete, a summative evaluation is performed to see if the stated goals 
of the project have been met. This is done through direct observation and public 
feedback solicitation, usually by an independent third party. If there are changes 
to be made with the project, a final, remedial evaluation is made to ensure that the 
intended knowledge transfer has been made. (Attachment B and Attachment C ac- 
companying this testimony illustrate an example of this evaluation process which 
we completed as part of our development of our permanent Earth science and dino- 
saur exhibit. The exhibit was produced with funding and content support from 
NASA).[Located in Appendix 2: Additional Material for the Record.] 

With educational programs, the process is similar. Educators are solicited for 
areas where an informal experience can add to the students’ understanding of the 
subject matter. When the programs are developed and delivered, feedback is given 
directly by the educators who bring their students to the center for the educational 
enrichment. These accepted evaluation procedures are required for National Science 
Foundation or National Institutes of Health grants and have been used by science 
museums for other federally-sponsored exhibits and programs. 
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Using the accepted evaluation practices mentioned above, NOAA, NASA and the 
Science On a Sphere users group (made up of all centers with a sphere installed 
as well as those centers where spheres are being installed), have embarked on spe- 
cific evaluation methods for the exhibit as well as the programs centered around the 
exhibit. Each funded NOAA project contains a detailed evaluation plan. NOAA 
asked the Maryland Science Center to lead a discussion of all SOS users on the dif- 
ferent SOS evaluation methods used to date and what method of prototyping and 
evaluation will best help science museums develop understandable exhibits and pro- 
grams for the target school group audiences and the general public. 

Using front end evaluation (a copy of the full evaluation can be found in Attach- 
ment A; see Appendix 2: Additional Material for the Record) of the Science On a 
Sphere exhibit — again conducted by third party evaluators — Maryland Science Cen- 
ter was able to implement and utilize the SOS exhibit in response to the feedback 
collected during the evaluation process. In our case, we developed specific staff-led 
programs to augment the SOS experience for our visitors. Overwhelmingly, the dis- 
play of the information and data, the quality of the presentation, overall appeal of 
the technology, and understanding of the purpose of the exhibit was extremely posi- 
tive. Once operational at Maryland Science Center however, our exhibits team no- 
ticed that when the SOS exhibit ran in auto-play mode using “canned” presen- 
tations, and no staff members were available to augment the presentation and an- 
swer questions about the data being presented, the level of engagement was short 
in duration. In general, when the SOS exhibit was facilitated by Maryland Science 
Center staffers, questions were answered, programming could be paused for expla- 
nation, and dwell time (time people spend at a specific exhibit) by visitors was very 
long in duration. When the SOS exhibit ran in auto-play mode — meaning the 
canned programs, with their taped narrative — the dwell time for visitors was far 
shorter. Visitors could not fully comprehend the auto-play presentations and moved 
on to other exhibit areas more quickly. “Canned” programming for SOS, absent a 
subject expert who could interpret the presentation for the casual enthusiast, was 
at too high a level. Programmers may have assumed too high a level of under- 
standing on the part of the museum-goer and the exhibit was losing audience as 
the visitor became confused or could not fully understand the presentation. 

In response to this, Maryland Science Center exhibit team members installed 
interactive computer kiosks around the SOS exhibit that offered a more basic inter- 
pretation of the imagery and programming being presented when the exhibit is in 
auto-play mode. Visitors can glean basic understanding of the programming’s more 
technical aspects by viewing a more basic interpretation on the interactive kiosk 
screen. Given this more basic knowledge, the visitor is given the tools necessary to 
gain a deeper understanding of the original intent of the more specialized canned 
programming. Program staff have also inserted more facilitated programs into the 
presentation schedule to engage more visitors more often using the SOS exhibit. To 
evaluate and measure the success of the remedial actions the exhibit team com- 
pleted a dwell time study of visitor interaction with the SOS exhibit prior to the 
installation of the interactive kiosks to create benchmark dwell time statistics. Now 
that the kiosks are in place, the dwell time study will be repeated and the data will 
be compared to the benchmarks created prior to the kiosk’s arrival. All the informa- 
tion gathered, the remedial actions employed, and the measures of success are being 
shared and reported to the NASA/NOAA led SOS users group so that the exhibit 
is as successful as possible at all locations around the country. As the installation 
of SOS exhibits began their roll out, greater collaboration between program creators 
and informal educators earlier on could have led to programs that did not need as 
much remedial modifications and augmentation. Partnering in the development 
stage may have gained SOS more audience and enthusiasm more quickly and in 
greater numbers. 

The case with SOS illustrates an example of how to improve scientific literacy. 
To improve the effectiveness of using informal education to help raise the level of 
scientific literacy in the United States, emphasis should be placed upon how non- 
classified information could be made available to the general public. Informal edu- 
cators such as ASTC members have the ability to dispense highly technical knowl- 
edge in a manner that the non-scientific public can understand. In the case of the 
Maryland Science Center, we employ a cadre of on-the-floor explainers, many of 
whom have received training from the various federal R&D mission agencies to aug- 
ment their own formal education. For every hour that we are open, we have staff 
members ready to engage our visitors to answer questions or offer ideas that stimu- 
late meaningful discussions about the subject areas. Our goal is to make gaining 
this knowledge engaging and fun, while showing how science and technology affect 
our daily lives. In the process, our hope is that we will excite and encourage some 
of our student visitors to consider careers in science and technology. 
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In the example provided about the installation, evaluation, and ongoing collabora- 
tion with the Science On a Sphere exhibit there is a working example of how federal 
agencies and science centers can better accomplish the goals of STEM education 
programs. We would encourage federal agencies to continue to expand ways that re- 
searchers and engineers collaborate with informal science education professionals to 
better engage the public. With SOS, scientists were made available to us, evaluation 
was encouraged, and NASA and NOAA sought our help in getting the message out. 
There was recognition that science centers, through their use of informal education, 
know how to engage visitors and spark their interest in the sciences. We know how 
visitors react and how best to present scientific discovery and scientific progress. 
And we know how to present it in ways that matter to them as individuals. 

The dialogue with science centers should be expanded — we want access to the 
knowledge and the discovery so we can distill, interpret and present it to the gen- 
eral public and school children in larger and more meaningful ways. We want to 
reach greater numbers of people more often so that scientific discovery becomes as 
much a part of a person’s everyday life as it can be. We believe that greater under- 
standing leads to greater acceptance that science is resident in everything we do — 
it doesn’t just happen in a laboratory. Science centers like the Maryland Science 
Center are a resource in every sense of the word and deserve to be viewed as such — 
from resources (financial and otherwise) to expertise and knowledge. We believe 
greater utilization of science centers as resources for federal R&D mission agencies 
is the best way help raise the level of scientific literacy with the general public, in- 
cluding school children. 
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Mr. Reiner also serves on the Board of the Maryland World Class Manufacturing 
Consortia and is a founding member and Treasurer of the Partnership for Balti- 
more’s Waterfront. In the past, he has served on the following Boards: Chester 
County, PA Chamber of Commerce, United Way of Chester County, Lackawanna, 
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Chairman Baird. Thank you, Mr. Reiner. I would mention that 
there is no need to apologize to this committee for providing addi- 
tional material and particularly evaluative. We appreciate the ef- 
fort you folks have done to evaluate your program effectiveness and 
value the information very much. 

Dr. Weiss. 
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STATEMENT OF DR. IRIS R. WEISS, PRESIDENT, HORIZON 
RESEARCH, INC. 

Dr. Weiss. Mr. Chairman, Members of the Subcommittee, thank 
you for the opportunity to participate in this hearing. 

My name is Iris Weiss, and I have spent the last three decades 
in research and evaluation in STEM education. I would like to 
share my thoughts on two issues, the first is how program evalua- 
tion can help the federal R&D mission agencies be more efficient 
and effective wherever they choose to focus their efforts to increase 
scientific literacy, and the second is where I believe these agencies 
should focus their efforts. 

To date the federal R&D mission agencies have not had a great 
deal of success in evaluating their STEM education programs. The 
same can be said for other federal agencies and for the broader 
field as well. 

How could evaluations be improved? First, the designs of pro- 
posed programs should be critiqued to determine if the interven- 
tions are likely to lead to the desired outcomes and how broad the 
impact would likely be, so programs could be improved before 
major costs are incurred. To take one example, the Department of 
Energy offered science teachers summer employment in their re- 
search labs. Program goals include deepening participating teach- 
ers’ knowledge of science and improving classroom practice at their 
schools. But a design critique would suggest that the program 
would be unlikely to achieve its classroom impact goal. Few teach- 
ers would have the time and expertise needed to develop student 
activities that were accurate, developmentally appropriate, and fea- 
sible to implement with the resources likely to be available, nor 
would the participating teachers be likely to have the time to help 
other teachers improve their classroom practice. A design critique 
might well have predicted what in fact happened. Teachers appre- 
ciated being involved in the program, reported that it deepened 
their understanding of scientific content and scientific research, but 
it did not have much of an impact on classroom practice. 

Similarly, formative evaluation of pilot programs would help the 
agencies be more efficient and effective in their STEM education ef- 
forts. At the pilot stage, the focus is not on impact but rather on 
whether the program can be implemented as intended, how it 
might be improved, or if it needs to be discontinued. 

There is no question that impact evaluations need to be improved 
as well, as the just-released report of the ACC makes clear. At the 
same time I believe the challenges associated with rigorous evalua- 
tions of education programs have not been adequately acknowl- 
edged in that report. In addition, effective evaluations require not 
only strong research design but also appropriate outcome meas- 
ures. Although developing instruments to assess teacher content 
knowledge and similar goals is not the responsibility of the federal 
R&D mission agencies, I believe that the lack of appropriate meas- 
ures will continue to hamper the mission agencies in efforts to in- 
crease their program effectiveness. 

Where should the federal R&D mission agencies focus their ef- 
forts to improve scientific literacy? Based on my understanding of 
the complexities of the K-12 education system and the expertise of 
these agencies, I believe they should play a relatively small role in 
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efforts to improve the formal K-12 education system and a larger 
role in the informal science arena. For example, current evidence 
suggests that teacher professional development is most effective in 
improving classroom practice when it is closely tied to instruction. 
We know that teacher content knowledge is necessary, but it is be- 
coming increasingly clear that it is not sufficient. Teachers also 
need to learn how to use their instructional materials well, how to 
figure out what their students understand and where they are 
struggling, and how to make appropriate instructional decisions 
based on that information. And teachers need opportunities to 
apply what they are learning in their own classrooms and to get 
constructive feedback. The federal R&D mission agencies certainly 
have the content expertise to provide professional development, but 
they have only limited understanding of K-12 education and they 
are not well-positioned to provide professional development that is 
practice-oriented and sustained over time. For greater and broader 
impact, rather than developing their own programs, I believe the 
federal R&D mission agencies should consider making scientists 
available to serve as content resources for local professional devel- 
opment, helping shore up a major weakness of many of those pro- 
grams. 

In contrast, I believe the federal R&D mission agencies are very 
well-positioned to make major contributions in the informal edu- 
cation arena along the lines we have just heard. Lack of coherence 
is not a problem here, as it is when we talked about curriculum 
development. In fact, having multiple pathways increases the like- 
lihood that a large number of people will benefit from the available 
resources. 

Thank you. 

[The prepared statement of Dr. Weiss follows:] 

Prepared Statement of Iris R. Weiss 

I appreciate the opportunity to testify before the Research and Science Education 
Subcommittee as it explores how the federal R&D mission agencies can contribute 
to improved scientific literacy for all students. There is no question that there is a 
wealth of scientific expertise in the various agencies, and considerable interest in 
helping to improve K-l2 STEM education. Moreover, much of the work of these mis- 
sion agencies focuses on areas that are of intrinsic interest to students, and can help 
motivate students both to engage in learning science and to consider STEM careers. 
With appropriate programs, carefully designed and well-implemented, the federal 
R&D mission agencies can both enhance levels of scientific literacy in the population 
as a whole and help ensure an adequate supply of well-qualified STEM professionals 
for the future. 

It is important to recognize, however, that there are many more “good ideas” (i.e., 
possible programs in areas of relevance to the agency’s mission that have the poten- 
tial to increase teacher knowledge, improve classroom practice, and enhance student 
knowledge and aspirations) than can possibly be implemented. There are substan- 
tial costs involved in designing, implementing, and evaluating new programs, and 
very limited resources available for these activities. Clearly there need to be criteria 
for deciding which of the many potentially good ideas should be implemented by a 
particular agency, and processes for deciding how to refine promising programs, 
which ones to scale up, and which ones to drop. 

The hearing charter makes clear that the goal is to increase the level of scientific 
literacy for all students. It is important, therefore, to consider the nature and scope 
of the K-12 education system that the mission agencies are trying to influence — 
50 states, more than 15,000 school districts, more than 100,000 schools, and millions 
of teachers responsible for STEM education, textbook publishers, test developers, 
etc. all making decisions that affect student opportunities to become scientifically 
literate. In addition, while there have been efforts to identify the core under- 
standings that constitute scientific literacy, the volume of content included in na- 
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tional and State standards documents is still much more than can possibly be ad- 
dressed in depth in the time available. Teachers and curriculum developers are 
faced with the unenviable choice of trying to cover it all, and doing so superficially; 
or taking seriously the recommendation for in-depth, inquiry-based learning, and 
leaving out some of the required content. 

In this context, current and potential programs have to be examined not only to 
see if they are effective in terms of adding value to the participating teachers/stu- 
dents, but also whether there are likely to be sufficient indirect benefits to a large 
enough number of students to make a meaningful difference in overall scientific lit- 
eracy. At present, the problem I see with many federal programs, including those 
of the R&D mission agencies, is that they have very limited potential for leverage 
and in some cases simply add to the confusion. 

How can program evaluation help federal programs be more efficient and 
effective in improving STEM education? 

The federal R&D mission agencies have not had a great deal of success in evalu- 
ating their STEM education programs; the same can be said for other federal agen- 
cies, and for the broader field as well. The natural desire to address the pressing 
problems in science education has taken precedence over the need to ensure that 
the investments will in fact have the intended impact. I believe that existing pro- 
gram evaluation tools and approaches can help increase the likelihood that STEM 
education programs supported by the federal R&D mission agencies (and others as 
well) will have a broad, positive impact. 

Evaluation is useful at various stages of a program. It can and should be used 
in (1) critiquing proposed programs to help make decisions about which ones to offer 
and to improve their designs; (2) monitoring program quality both to allow appro- 
priate mid-course corrections and determine if the program is ready for rigorous 
evaluation; and (3) assessing program impact. At present, it appears that some of 
the tools and approaches that evaluation has to offer are used some of the time in 
some of the STEM education programs supported by the federal R&D mission agen- 
cies; their more consistent application would help improve the quality, impact, and 
cost-effectiveness of the agencies’ efforts to enhance overall scientific literacy. 

Evaluation as design critique 

In terms of program design, the first step any agency needs to take is to identify 
needs relevant to their mission and expertise. The federal R&D mission agencies 
have been very successful in this regard; virtually every program they offer can be 
readily mapped both to the mission of the agency and to the needs of the designated 
target audience(s). 

But targeting an appropriate need does not necessarily mean that the programs 
are addressing priority needs; one can assume that at least some students and 
teachers lack knowledge in any given area of science, and that many more students 
and teachers are likely to lack knowledge in areas at the cutting edge of science. 
Since any program aimed at increasing teacher or student knowledge could be justi- 
fied by making the case that it addresses an existing need, simply being able to 
demonstrate need is not an adequate criterion for making decisions among potential 
programs. Given scarce resources, agencies need to be able to decide which of the 
many needs that are consistent with their mission are the most important to ad- 
dress, and which of those they have the capacity to address well. Only then does 
it make sense to move ahead with program development. 

Not having been part of the program planning discussions, I can’t tell the extent 
to which the mission agencies’ STEM education program rationales were made ex- 
plicit and the various priorities debated. But my impression from the multitude of 
topics, grade ranges, and approaches the various agencies are using is that decisions 
have been made based on whether a particular idea was of interest to someone in 
a decision-making position, rather than whether the program was part of an overall, 
coordinated strategy for maximum leverage on K-12 education. 

Even more important than whether a program targets a priority need is whether 
the proposed intervention is likely to have the desired impact; no matter how impor- 
tant the need, ineffective programs are a wasted investment. Conducting a “design 
critique” of a proposed program can help improve the design, or in some cases lead 
to a decision not to go forward with programs where the odds are stacked against 
them. And the very good news is that design critiques are not an expensive under- 
taking; they require only modest amounts of time from people who understand both 
the system that is being targeted for improvement and what has been learned in 
prior efforts. 

We need to pay more attention to the fact that STEM education programs that 
either have little likelihood of impact, or will impact only a small number of teach- 



36 


ers/students, are not going to make much of a difference in overall scientific literacy. 
Again, the criterion of likely impact based on prior research and the “wisdom of 
practice” seems not to have been uniformly applied in the STEM education pro- 
grams offered by the federal R&D mission agencies. 

To take one example, the Department of Energy has at various times offered 
science teachers summer employment in their research labs, an expensive under- 
taking given the costs of salary, transportation, and lodging. The goals of the pro- 
gram have been to deepen participating teachers’ knowledge of science, and to im- 
prove instruction not only in the participating teachers’ classrooms, but in those of 
their colleagues at the school as well. 

Developing a “logic model,” a standard tool in program evaluation, would have en- 
abled the designers of that program to see that there were major holes in the pro- 
gram’s theory of action, places where the links between activity and impact were 
weak at best. One could readily make the case that teachers would learn more 
science, and learn more about scientific research, by being placed in a research lab- 
oratory. However, the science content teachers were learning was likely to be well 
beyond what their students would be expected to learn, and they would not have 
the sophisticated equipment needed to carry out the investigations. Few teachers 
would have the time and expertise needed to develop instructional activities to make 
the activities developmentally appropriate for their students and feasible to imple- 
ment with the available resources; nor would participating teachers have the time 
to help other teachers apply what they had learned. Thus, while teachers who par- 
ticipate in these kinds of programs often report that they gained a great deal from 
these experiences, it should not be surprising that the improved classroom practice 
that was a major goal of the programs rarely materialized. In this case and many 
others supported by federal, state, and local agencies, considerable resources have 
been devoted to programs where lack of classroom-level impact could have been an- 
ticipated. 

Formative evaluation to enable mid-course corrections and determine if programs are 
ready for rigorous evaluation 

Given the start-up costs associated with the development of any new program, it 
make sense to fund only those that have great potential to begin with, and then 
based on the lessons learned during implementation to refine the programs to get 
the kinks out. Evaluations of the mission agencies’ STEM education programs would 
also be improved by more systematic attention to monitoring the quality of program 
implementation and use of the resulting feedback. 

From an external perspective, the fact that some initiatives have been modified 
over time suggests that at least some of the federal R&D mission agencies employ 
formative evaluation strategies for at least some of their STEM education programs. 
It is less clear whether the STEM education programs supported by the federal 
R&D mission agencies use evaluation for quality control purposes when programs 
are expanded. Often the people who design a program, e.g., for teacher professional 
development, are able to implement it well, but when the program is expanded the 
quality tends to suffer. It is important both to monitor initial program implementa- 
tion and fine-tune the design as needed, and to monitor the quality of implementa- 
tion during scale up. Ideally, evaluations of the quality of implementation would in- 
clude observations of program activities by people who have expertise in both con- 
tent and the target populations; interviews with key stakeholders, including in 
many cases students, teachers, administrators, and parents. Often it is appropriate 
to collect some interim data on impact to see if the design needs to be fine-tuned, 
or additional support provided to program implementers. 

Sometimes a preliminary evaluation provides evidence that a program is unlikely 
to achieve its goals, so a more rigorous evaluation is not necessary. For example, 
my organization was once asked to evaluate a statewide program that had the goal 
of “transforming elementary science education.” One of the primary interventions 
was having STEM faculty visit classrooms — typically once a semester — and model 
for teachers how to conduct science demonstrations. The client wanted evidence to 
see if this strategy was paying off in terms of improved classroom practice. Recog- 
nizing the limitations of survey self-report data, they asked that we do classroom 
observations, which would have required site visits to a fairly large number of treat- 
ment and comparison classes, clearly an expensive undertaking. 

From our perspective, finding out that something that could not possibly work in 
fact did not work seemed to us to be a poor use of both our time and taxpayer 
money; we convinced the client to let us interview a small number of teachers before 
committing to a more extensive evaluation. Teachers told us that (1) they were 
happy to have scientists visit their classrooms because the kids enjoyed it and got 
a better sense that scientists were like most people, not nerdy beings in laboratory 
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coats; (2) they thought it would be a good idea if they did demonstrations like the 
scientists had done, but acknowledged that they rarely did so — they didn’t know 
whether the demonstrations would “work;” they didn’t have the necessary materials; 
and they were concerned that they wouldn’t be able to answer questions students 
raised. In this case we were able to convince the client to forego a rigorous evalua- 
tion, but not, unfortunately, to revamp the clearly ineffective program. 

As another example, if materials have been developed for classroom use, but ini- 
tial evaluation data show that teachers aren’t using the materials because of they 
do not appear to be well-aligned with state standards, time and effort spent doing 
a careful evaluation of impact on student learning would not be warranted. Given 
the substantial costs involved, only programs that have a reasonable likelihood of 
substantial impact and can be implemented well should be subjected to rigorous 
evaluation. 

It is particularly important to provide incentives for agency personnel to use eval- 
uation feedback for program improvement, rather than allowing people to continue 
to implement poorly designed, inadequately implemented, or ineffective programs. 
Unfortunately, there appear to be pressures at every level of the system for people 
to overstate the success of their programs, highlighting positive aspects and glossing 
over problems, which may help explain the observation that almost everything ap- 
pears to work, but nothing much changes. 

Summative evaluation to assess program impact 

What most people mean by program evaluation has nothing to do with design cri- 
tique or studying the quality of implementation; rather evaluation is typically 
equated with an assessment of the impact of a particular activity or set of activities. 
It is important to recognize that rigorous evaluation is very difficult, and it is there- 
fore not surprising that the federal R&D mission agencies have encountered many 
challenges in assessing the impact of their STEM education programs. First, as a 
profession, we lack instruments to measure many of the outcomes we care about. 
For example, many STEM education programs over the last several decades have 
aimed to deepen teacher content knowledge, but until recently there were no instru- 
ments of demonstrated validity and reliability that were feasible for use on a large 
scale; even now such instruments exist for only a few topics. As a result, program 
evaluations have had to depend on notoriously suspect measures, such as asking 
teachers if they thought their content knowledge had improved! Programs targeting 
student knowledge have faced similar problems, as it has proven difficult and costly 
to develop measures of conceptual understanding; existing instruments are more 
likely to assess student knowledge of vocabulary or the apocryphal n steps in the 
“scientific method,” rather than the in-depth understanding sought by STEM edu- 
cation programs. 

Even if appropriate measures were available, program evaluation has to navigate 
many other difficult challenges as well. Much attention has been paid of late to ran- 
domized field trials as the “gold standard” for evaluating program effectiveness. 
There is no question about the value of this approach, but there are many questions 
about its cost and feasibility. (It is particularly ironic that at a time when school 
districts are very interested in “research-based” programs, they are reluctant to par- 
ticipate in research because of the many pressures they are dealing with.) 

And as the recent report by the American Competitive Council notes, decisions 
about education policy and practice shouldn’t be based on single studies, however 
well-designed. To be most helpful, an evaluation of program effectiveness should in- 
clude multiple studies to answer question not only about whether the program 
achieves its desired outcomes, but also with whom and under what conditions. Fi- 
nally, summative evaluations need to determine if programs have had unintended 
negative consequences. 

Where should the federal R&D mission agencies focus their STEM edu- 
cation efforts? 

Based on my understanding of both the expertise of the federal R&D mission 
agencies, and the complexities of the K-12 education system, I believe these agen- 
cies should play a relatively small, supporting role in efforts to improve the K-12 
education system, and a more direct and major role in the informal science arena. 

I suspect that was not the advice I was expected to provide, as I was asked to 
use what we have learned from research to make recommendations for the develop- 
ment of programs for pre-service and in-service STEM teachers. (Before I explain 
my reasoning, I would like to point out that the research I and others have con- 
ducted on effective professional development has not progressed as far as one would 
hope, for a myriad of reasons. I already mentioned the lack of valid and reliable 
measures of teacher learning that are feasible for large-scale administration. In my 
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view, it is both appropriate and essential that the Federal Government support such 
development efforts, as the private sector has few if any incentives to undertake this 
difficult and expensive work. But that is probably an appropriate task for the Na- 
tional Science Foundation rather than for the federal R&D mission agencies that 
are under consideration in this hearing. A second major problem has been the lack 
of a system to help ensure the steady accumulation of knowledge in key areas such 
as professional development for STEM teachers, again a challenge more for NSF 
than for the federal R&D mission agencies. Much of what we “know” about effective 
professional development is based on the insights of expert practitioners, rather 
than on clear empirical evidence. Richard Elmore has characterized the emerging 
consensus not as a substitute for research, but as a set of sensible propositions that 
can be used to guide practice and as hypotheses to be tested.) 

Although the research is far from definitive, the emerging consensus in the field 
is that professional development is most effective in changing classroom practice 
when it is closely tied to classroom instruction. Although there is no question that 
teacher content knowledge is necessary, it is becoming increasingly clear that teach- 
er knowledge of content is not sufficient. Teachers also need to learn how their in- 
structional materials can be used to help students learn science concepts; how to 
figure out what their students understand and where they are struggling; and how 
to appropriate instructional decisions based on that information. Teachers also need 
opportunities to apply what they are learning in their own classrooms; to share 
their struggles and triumphs with other teachers; and to get feedback they can use 
in improving their instruction. To be effective, it appears, professional development 
programs need to be intensive, extensive, and sustained over time. 

The federal R&D mission agencies clearly have the necessary content expertise, 
but they have only limited expertise in improving classroom practice. Thus they do 
not appear to be well-positioned to make a substantial contribution to teacher pro- 
fessional development of the nature and scale needed to increase overall science lit- 
eracy. A number of the federal R&D agencies have offered professional programs for 
many years, but those programs typically reach only small numbers of teachers, in 
many cases “volunteers” who tend to be already relatively strong in content knowl- 
edge. To be effective in providing professional development, the mission agencies 
would need to create mechanisms to be able to stay current about what is being 
learned about effective professional development, and apply that knowledge to their 
professional development programs. And they would have to develop and maintain 
on-going relationships with a sufficient number of districts to make much of a dif- 
ference. 

In my view, rather than having the federal R&D mission agencies develop and 
implement their own professional development programs, it would make sense to 
have agency scientists available to serve as content resources for local professional 
development. It would also be helpful if agency scientists were available to assist 
organizations engaged in the development of professional development materials for 
more widespread use. 

Similarly, I would not recommend that the mission agencies continue to develop 
instructional modules for classroom use. That is not to say that the materials the 
federal R&D agencies have developed are of poor quality, but rather that the K- 
12 education system lacks incentives for teachers to find those materials, or once 
found, to use them in their classrooms. Many science teachers are already hard- 
pressed to address the content included in state standards in anywhere close to the 
depth needed to develop student understanding, so adding in supplemental activi- 
ties may be a difficult sell. In fact, having the mission agencies provide activities 
for classroom use can actually have a negative effect, adding to the incoherence in 
the system as different teachers make different decisions about what to leave out 
in order to make room for these activities. The teacher of the next course may well 
have some students who have engaged with the topic as addressed in the “regular” 
materials, some with the supplemental activity, others with both, and still others 
with neither. In that situation, teachers can’t win no matter what they decide to 
do. 

Just as serving as content resources for others engaged in professional develop- 
ment would be helpful, in my opinion the federal R&D mission agencies can con- 
tribute to the improvement of the K-12 education system by making relevant data 
accessible to people who develop curricula, assisting them in understanding their 
potential not only for engaging students but also for helping them learn important 
content as outlined in national and State standards. 

In contrast to the cautious approach I recommend for involvement in the formal 
K-12 education system, I believe the federal R&D mission agencies are well posi- 
tioned to make major contributions in the informal science arena, e.g., through the 
development of interactive exhibits for science centers on phenomena of interest to 
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students, parents, and the general public; speakers’ bureaus; activities for after- 
school programs; newspaper inserts; television programs, etc. 

Informal science education vehicles can also be used by the mission agencies to 
help ensure an adequate science pipeline, for example disseminating information 
about science career opportunities requiring different levels of education. The fed- 
eral R&D mission agencies can sponsor programs for interested students to interact 
with scientists, with special efforts to encourage participation of students from 
under-represented groups. Other efforts could target parents, to help ensure that 
their children keep their options open by enrolling in elective mathematics and 
science courses. 

While coordination of efforts among agencies to avoid unnecessary duplication of 
either infrastructure or resources is appropriate, lack of coherence is not an issue 
as it is in the formal K-12 education system. Different people will access different 
resources in different ways and at different times; having multiple pathways in- 
creases the likelihood that people will benefit from the available resources. 
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Discussion 

Chairman Baird. Thank you, Dr. Weiss. Fascinating series of 
perspectives from all of you, and I am grateful for your insights. 
I will begin the questioning and then in five minutes or so we will 
yield to Dr. Ehlers. 

One of the terms that came up repeatedly in various testimony 
was outcomes and the importance of looking at outcomes. What 
would each of you — and I will let each of you take a shot at this — 
what would be the most important outcome that you think could 
derive from participation by the mission agencies in the education 
endeavor? If you had to measure it — and I recognize frankly some 
of it is rather ephemeral and may be difficult to quantify and I re- 
spect that. Set aside the issue — don’t define the goal as something 
measurable, define something desirable first and we will worry 
about measure in a second. What would you think are the most im- 
portant, left to right, Ms. Froschauer? 

Ms. Froschauer. One of the things I believe that would help 
classroom teachers the most, and that is my perspective, is re- 
search that would provide us with information concerning how best 
to teach concepts to students so that they truly can conceptually 
develop the ideas. The research base in many areas is lacking, and 
expanding that research base would be very valuable. And most of 
that can be done quite well with some longitudinal studies and look 
at how students actually learn over a long period of time and add 
to their conceptual understanding. 

Chairman Baird. Mr. Lach. 

Mr. Lach. I think what would help most is having the federal 
R&D mission agencies measure the way that they connect students 
and parents and teachers to the practice of science through their 
laboratories and their facilities and the way that they inspire that 
same population to get excited about the world of science and its 
practice, focus on the informal side. 

Chairman Baird. Dr. Nelson. 

Dr. Nelson. I would like to see an outcome that would allow the 
mission agencies to be full partners with the schools and the com- 
munity colleges and the programs that help prepare not necessarily 
the very high-end, top 10 percent students who are going to be sci- 
entists and engineers, but help the forgotten majority of the stu- 
dents below that who are very necessary. We need three or four or 
five good technicians for every engineer we have got in the field. 
Those students need to be both excited to participate in a career 
like that, prepared through a good K-10 kind of preparation in the 
schools, but where the agencies can help is to take the next step 
through their last couple of years of high school into community 
college to prepare them to work on the cutting edge in the mission 
agencies as high-paid, well-prepared technicians and then support 
personnel. 

Chairman Baird. Mr. Reiner. 

Mr. Reiner. In my, or our arena, if you will I think the mission 
agencies have a role to play in terms of exciting not only the stu- 
dents but their parents and the general public because I think that 
we need to improve science literacy, we need to have people under- 
stand that science is all around them. 
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I have a couple anecdotes that I think apply to that. Dr. Tom 
Jones, a former Shuttle astronaut, has told me that he first became 
interested in astronomy when he looked through the telescope at 
the Maryland Science Center as a young child. Also, the current 
doctor in charge of the Baltimore Public Health System remembers 
going to a science summer camp at the Maryland Science Center 
where he dissected a cow’s eye, and that gave him his first example 
of medicine. 

So we have that opportunity, I think the federal R&D mission 
agencies can help us in terms of getting people, the public, stu- 
dents, to relate to the field. 

Chairman Baird. I notice in those comments the absolute cen- 
trality of hands-on experience which I know your facility is — just 
really the hallmark of your facility and in both those anecdotes, it 
was a hands-on experience that excited someone. 

Mr. Reiner. There is nothing like the face of a student who, in 
interacting with an exhibit, suddenly gets it. 

Chairman Baird. Dr. Weiss. 

Dr. Weiss. One outcome would be student interest in science, the 
wow factor that you referenced. A second would be general science 
literacy. We have measured that over time, and it has been pretty 
disheartening what the results have been. And if the agencies 
choose to work in the formal system, then I would say the outcome 
would be improvement at scale, for example, having science text- 
books used by millions of kids have relevant applications as op- 
posed to just a small number of people benefiting. 

Chairman Baird. The scale issue also is something many of you 
mentioned, and I think that is a central question: How do we scale 
this up? It is terrific if 100 teachers or 50 teachers or even only 
20 can go to a summer workshop, but how do we scale it to reach 
the forgotten majority that Dr. Nelson alluded to? Maybe we will 
be able to get to that question in a moment. I would just conclude 
that we had a workshop out in my district with NSF and a bunch 
of teachers, and someone asked at one point what do you think the 
goal should be; and for me, the essence is, as a science teacher my- 
self actually before this job, is wonder and discipline. You said the 
wow factor. I want somehow, and I think what the agencies can do 
is the wonder part. There is nothing like talking to a Space Shuttle 
astronaut or somebody under the ocean or dissecting a cow’s eye. 
That wonder part is critical but the discipline part that helps them 
understand it takes some rigor to answer these questions. 

And with that, let me yield appropriately to a gentleman who 
knows first hand about that, the Ranking Member. I would also 
note the presence of Eddie Bernie Johnson, former Ranking Mem- 
ber of this subcommittee and a valuable asset to it. Thank you. Dr. 
Ehlers? 

Mr. Ehlers. Thank you very much. I really appreciate the testi- 
mony. I am a hands-on person as an experimental physicist, and 
I am a great believer in hands-on education as well. I am just curi- 
ous, of the — you know, we are talking here about agency programs, 
agency ideas. How much of them are directed to the high-school 
student and how much to the elementary school student? Let me 
just get an idea from each of you what you think it is, let me say 
the ratio of high-school to elementary school? Ms. Froschauer? 
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Ms. Froschauer. I don’t know if I can give you an exact figure 
on that, however 

Mr. Ehlers. I didn’t expect exact figures. 

Ms. Froschauer. — there is a great deal more emphasis on high 
school than there is on elementary school which we know needs 
to — we need to have a shift in that. There needs to be more empha- 
sis on elementary school because as so many people have ex- 
pressed, when you are beginning in their education, we want to ex- 
cite students so they actually consider taking more science as they 
go through the K-12 system. 

Mr. Ehlers. Michael? 

Mr. Lach. [no response], 

Mr. Ehlers. Okay. Dr. Nelson. 

Dr. Nelson. I think there has been — it moves around. It is a 
moving target. I think recently people have been focused on middle 
school, you know, which has been kind of a great wasteland of edu- 
cation in terms of what the focus and the coherence of the pro- 
grams there, so it has been getting a lot of attention. My impres- 
sion is it is kind of spread evenly across the board. 

Mr. Ehlers. Mr. Reiner. 

Mr. Reiner. I think the mission agencies would like to focus on 
high school. In our case we force them to tailor their content or we 
tailor the content to upper elementary and middle school. 

Mr. Ehlers. Okay. Dr. Weiss. 

Dr. Weiss. I haven’t a clue. 

Mr. Ehlers. Okay. That is a perfectly valid answer. I don’t ei- 
ther, otherwise I wouldn’t have asked the question. Years ago I 
proposed, and unfortunately that part of the bill got removed as it 
went through the process. This is when we still had Eisenhower 
funding and we had a clearinghouse in Columbus. I proposed that 
that clearinghouse be charged to have a listing, an Amazon.com 
type of listing of all of the different units available from all the dif- 
ferent public agencies, from the corporations, the chemical society, 
et cetera, et cetera. And by Amazon.com style I mean teachers who 
used a particular unit would send back their evaluations, you 
know, one star up to five stars and tell other teachers how they 
have used it so a high school chemistry teacher wanting to teach 
something about the gas laws would just go to that website, punch 
in gas laws. There might be 20 units that would fit, should read 
the evaluations, download the best one for her class the next day. 
Unfortunately, as I say, that got lost; but I still think it would be 
very useful. 

The other thing it seems to me would be useful in terms of the 
government agencies is some sort of STEM czar, and I don’t mean 
that literally but something that coordinates all the different pro- 
grams because we have an incredible hodgepodge out there; and 
how is a teacher to sort them out? How do they relate to each 
other? How can you effectually use them in the curriculum? It 
seems to me that having the soliciting I talked about plus some co- 
herence to the Federal Government’s efforts might be very bene- 
ficial for all teachers. 

Finally, let me just make a pitch. In picking up on what Mr. 
Reiner said about getting kids excited. I really think you have to 
start in the elementary school very strongly. If we are going to get 
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the type of technicians we need, and I always say the jobs of the 
future require an understanding of the basic principles of math and 
science. I mean, I think that is pretty self-evident. So how do we 
convey that to the kids? I think you have to start in preschool al- 
ready, emphasizing these ideas. And I am pleased I just managed 
to get attached to the Head Start bill last week. The Head Start 
programs also have to deal with what we call math readiness and 
science readiness, just learning simple skills of classification and 
enumeration, things of that sort. If we don’t get them started early, 
they are not going to do it in middle school. If they don’t do it 
there, they don’t do it in high school, they get to the university and 
it is too late. They have to spend six years if they want to become 
an engineer. So I think it is crucial that these programs that we, 
collective we, develop for use in school be able to span the spectrum 
and really develop the interest. 

With that I will yield back. 

Chairman Baird. I will yield to Dr. McNerney in one moment 
but I would like to give the panelists the chance to respond to the 
particular issue Dr. Ehlers raised which seems intriguing to me. Is 
there such a clearinghouse as he has described and if not, would 
it be useful to you and in what way would it be most useful? 

Ms. Froschauer. There isn’t a single clearinghouse for all mate- 
rials that I have ever heard of. There have been some attempts to 
have clearinghouses for materials, and currently there is nothing; 
and it is beneficial to have a clearinghouse. 

Something else that is interesting is that currently when pro- 
grams come out of an agency such as NSF, there can’t be any kind 
of a rating system coming from NSF. Actually, they can’t put their 
good NSF stamp of approval on it and say this is what everyone 
should be using. And so it even makes it more of a dilemma as to 
what really quality programs are out there for teachers to pull 
from. 

Chairman Baird. Mr. Lach. 

Mr. Lach. There is not a clearinghouse that I know of other than 
a web search which I know many of our teachers use. I would sort 
of add to the discussion, one of the sensibilities we have learned 
in Chicago is that just using curriculum by itself is necessary but 
not sufficient to get where we need to go. I spend an awful lot of 
time and energy connecting well-designed curriculum to workshops, 
to in-school coaching, to assessments, to leadership development 
work, to grade-to-grade, school-to-school sequencing; and I think 
that part is really, really important and I would want to make sure 
any such clearinghouse really highlighted the connections between 
all the things that a teacher has to worry about. If we make it just 
a place to download a PDF of a gas law experiment, it is not going 
to be — it is necessary but it is not sufficient to get the kind of 
change we need. 

Chairman Baird. Other comments on that? 

Mr. Reiner. On the informal science education, the members of 
the Association of Science Technology Centers do share successful 
programs; and the website of ASTC is a place that you can go. Ad- 
ditionally, when we receive an NSF or NIH grant, we are required 
to share any findings we get with other member institutions, but 
I know of no other clearinghouse. 
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Dr. Weiss. I would like to comment on that as well. Some teach- 
ers can pull together excellent materials and organize them into a 
coherent curriculum, but most teachers have neither the time nor 
the capacity to do that; and in our research, when teachers have 
been faced with more than they can cover in the time available, 
they make choices. But the choices tend to be based on what they 
believe is engaging to their students, and a lot of the prerequisites 
get lost, the coherent whole of children getting an opportunity to 
learn important science goes away. 

So the kind of clearinghouse approach I would recommend would 
be to make these “wow” factor types of things available to cur- 
riculum developers so they can get into the system at scale as op- 
posed to through the work of individual teachers. 

Chairman Baird. Dr. McNerney. 

Mr. McNerney. Thank you, Mr. Chairman. One of the big chal- 
lenges we face is that STEM education is hard work. It is not easy 
to get a degree in math or physics or engineering. It takes a lot 
of hard work. And so part of our job is finding out what it is going 
to take to inspire this coming generation to do that hard work, to 
get involved and instead of going to the frat party that they want 
to go to, to actually do the work and this starts when they are 
young. This sort of drive to achieve something in science or engi- 
neering starts when they are young. 

Dr. Van Reiner, you had mentioned that you have scientists 
come in to your science center. I am wondering, what is the most 
effective thing in your observation to get kids wowed, to get kids 
excited and inspired about science? Is it direct interaction with sci- 
entists or is it hands-on? What works the best? 

Mr. Reiner. At the early ages, it is definitely the hands on. As 
they get into upper elementary or middle school, it is a combination 
of the hands on and the direct interaction. I can remember we had 
a USGS geologist, a young woman who was explaining what she 
did for her job; and this young seventh-grader said to her, you real- 
ly like what you are doing, don’t you? And she said yes. And they 
pay you for that? So I think that is — I have got a thousand anec- 
dotes. But I do believe that it is important for the scientist to have 
a face-to-face time with the upcoming generation if you will in 
order to be able to practice explaining things in everyday language. 

Dr. Nelson. I would like to comment on that, too. I think one 
of the issues of the pipeline that we don’t talk about very often that 
is really important is we need to focus on getting kids in the front 
end of the pipeline. That is certainly true. But one of the things 
I have found in working with lots of students and trying to con- 
vince them that they might be interested in being scientists or en- 
gineers or going to work and even being science teachers is this no- 
tion that the pull on the other end of the pipeline isn’t that strong. 
A lot of times people say, well, why should I be an engineer? It is 
not such a great job anymore. How attractive is it to get my Ph.D. 
in biology when I could get my business degree and become a post 
doc for the next eight years, and when I am 40 I might get an as- 
sistant professorship job? So I think we need to work on both ends 
of the pipeline to make the jobs for students very appealing, and 
the agencies can certainly work on that at the front end so that 
students see this as a possibility. 
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Think back in the ’60s when you were going through school and 
the community, the government, and everybody else was paying 
people to go to graduate school. The universities were booming, ev- 
erybody who graduated could get a job right away. And now we 
have this huge pool of post docs and others out there. So the sys- 
tem is different today, and I think that does have an impact. 

Mr. McNerney. I will yield back. 

Chairman Baird. Ms. Johnson. 

Ms. Johnson. No questions. 

Chairman Baird. Okay. We will go to a second round. This is 
very interesting, and I appreciate it greatly. 

Let us move to the issue of scale a little bit because that is some- 
thing that many of you mentioned and a common thread and also, 
Dr. Nelson, you talked about the challenge of — I want to sort of put 
out two issues, one is the issue of scale and the second one is the 
issue that Dr. Nelson talked about, about NASA proliferating edu- 
cational material, posters, pamphlets, et cetera, but lacking some 
of the direction, the kind of things that Dr. Weiss maybe men- 
tioned. So I am going to throw out those two topics and open it up 
to any of you about either of those topics that you want to address, 
either how we scale things up or how we watch out for this pro- 
liferation of materials that may be well-intentioned but not well- 
targeted. Mr. Lach, you have got direct experience with some of 
this, a big-scale system. 

Mr. Lach. I know a thing or two about scale. And I think that 
is really the key point. A lot of what we have done in Chicago has 
been based on Iris’ work among others, and I think what we found 
out is we may not have all the answers but we know an awful lot 
more about how to leverage pretty dramatic change in a large sys- 
tem. It involves coherence, it involves an intense focus on capacity 
building, it means connecting instructional materials to assess- 
ments, to coaching, to support, it means focusing on leadership, 
and it means pulling everybody in the community together, all the 
museums, the universities, the labs, you know, to work on this to- 
gether. It takes a long time and it takes an awful lot of work. 

Chairman Baird. How did you find time for the people and par- 
ticipating to do this? How did you get their buy-in in that system? 

Mr. Lach. Our work in Chicago began through a series of NSF 
systemic initiative grants. It took us several years to sort of figure 
out how to use them and how to use them well. And we began with 
a — we knew that need. That need was very clear. We began with 
a vision that said this is how we are going to move things ahead. 
We had 87 different math curriculum in Chicago at the K-8 level 
when we started. Now we are down to two at the K-5 level. In a 
local-controlled district that takes an awful lot of convincing and 
cajoling to do. But other groups would sort of come along once they 
would see that sort of coherence. And it means, you know, tending 
to your stakeholders. I spent a lot of time and energy making sure 
our friends at universities, our friends at labs and museums under- 
stand the role that they are to play; and we do a lot of back and 
forth to make sure that that makes sense. 

Chairman Baird. What is your portfolio? Did someone crown you 
as the science education czar and give you some authority or is it 
just your persuasive personality that gets you through the day? 
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Mr. Lach. I was a classroom teacher and had enough of a loud 
mouth that was sent to Washington to be a fellow, and then I 
learned an awful lot from Congressman Ehlers who sort of gave me 
a perspective about scale and about policy that I just didn’t have 
at the classroom level. And then when I came back to Chicago, it 
took a little bit of time but they had me in charge of science and 
then science and mathematics. 

Chairman Baird. And you are given that respect. People ac- 
knowledge that is your role and they look to you for help and you 
get to give some guidance and governance I guess? 

Mr. Lach. Yes. I pause because I think — and this is a little bit 
of an aside. I think one of the things we are learning is that in- 
struction and leadership in mathematics and science is really, real- 
ly important; and it doesn’t exist very much in the educational sys- 
tem. There is lots of pretty compelling research now that shows 
principals and school leaders, when they lead, the practice of lead- 
ership differs around leading and around mathematics and around 
science; and that is not something most of the people — most local 
and state education agencies understand and I don’t think the sys- 
tem quite addresses yet. And it is a really important factor if we 
are going to leverage the kind of changes we need. 

Chairman Baird. Well, we appreciate the work you do, and I am 
also very grateful to hear that an NSF grant was used so well. You 
may be interested to know in our NSF reauthorization bill which 
Dr. Ehlers and I wrote together we have actually lengthened the 
time period for some of these demonstration projects that had been 
three years and you are out, basically. So that right when you get 
things where you have tweaked it enough to think it is finally 
working, you get the first class of students through, then it runs 
out. We are actually going to make that longer. It is also nice to 
know that somebody who has worked for Dr. Ehlers has gone on 
to do some great things. Not a surprise at all. You learned from 
the feet of the master. 

Dr. Ehlers is recognized. 

Mr. Ehlers. I am not sure how I could follow that up. Let me 
just say that when Michael spent a year in my office, he did a bet- 
ter job of learning how Congress operates than any fellow I ever 
had. He had an instinctive approach. And I think what really hap- 
pened when he got back to Chicago is he realized that if anyone 
can understand how Congress works they can figure out how the 
educational system works. 

I think it wasn’t a factor also that you arrived there shortly after 
the Chicago Public Schools were so-called privatized and that re- 
sulted in much greater centralization of authority and power, and 
that is something you could leverage. Is that correct? 

Mr. Lach. Yes. 

Mr. Ehlers. So it was a combination. But you have done a beau- 
tiful job there. Let me also answer one other question that I heard 
raised at some point and I don’t recall where or how but that was 
about how one propagates this. And the best example of that I have 
seen is the American Meteorological Society offers a summer pro- 
gram for teachers at I think various grade levels. But one require- 
ment of signing up for that course, and it is a very good course, 
it is about a month or more and the teachers are paid for their ex- 
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penses, et cetera — I think there is a stipend — but one requirement 
is that every teacher, when they go back to their school system, 
have to set up workshops to teach 10 other teachers the same ma- 
terial. And then those teachers have to make a commitment to 
propagate it to their own school buildings. And so in a short time, 
the AMS curriculum went from being just taught to a workshop 
and ended up with 100,000 teachers using it. And you know, if 
somebody looked at that, perhaps it is easier than most to develop 
a concise unit. Kids can study clouds and weather and so forth so 
you know, there is no expense involved with equipment. But never- 
theless, I thought it was a brilliant idea and something that we 
might pursue as well. 

I don’t have any other questions on my mind at the moment. I 
just very much appreciate the breadth of experience represented 
here and the comments that you have made. It has given me a lot 
of insight of what we should be trying to do legislatively as well 
and to take into account the concerns you have raised. But I don’t 
see any way we are going to break it without, as Michael has indi- 
cated — I believe our emphasis has to be professional development 
because before I came here I worked with a lot of schools and a 
lot of teachers; and I found the teachers wanting to teach science 
well, wanting to teach math well, but many were scared because 
they didn’t know the subject material. Many others didn’t really — 
they weren’t scared but they didn’t know how to tackle it and do 
it right. And professional development is the only way you are 
going to deal with that, and that is why the Eisenhower program 
was a good thing, even though it didn’t always do it well. But that 
is something we lost in No Child Left Behind only because the 
mechanisms are there but the money was never provided. And so 
we actually lost something going from Eisenhower to No Child Left 
Behind. 

With that I will yield back. 

Chairman Baird. Did any others want to comment on the issue 
of scale or the other couple of questions? 

Dr. Nelson. I would like to make a quick comment on that, and 
it relates to professional development, too. One of the reasons I 
went back to the university that I went to, Western Washington 
University, is that it is a former normal school. It prepares roughly 
500 teachers a year, and with the bulge of baby boomers moving 
through the system now, we have a real opportunity to provide the 
system with new teachers, with young teachers, with new ideas. 
And one of my stated goals at the university is that I have been 
at this for five years now. I am giving myself another five years 
to say in that time I hope to be graduating teachers from this insti- 
tution who don’t need remedial professional development. We can’t 
always continue at the same model of assuming that the teachers 
in the schools are not at the level we want them to be. And so we 
are working very hard to try and graduate new teachers who know 
how to choose and use the best curriculum that is out there, who 
know what good instruction looks like and can have a beginning at 
least level of practice for that but also really importantly know how 
to collaborate with their peers and to partner with us at the uni- 
versity and others to improve their instruction focused around the 
performance of their students. 
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So now I am starting to focus on not just teachers, but the ad- 
ministrators are such a huge role. We are finding in our NSF grant 
that probably the biggest factor on whether a school is making real 
progress in improving their science education programs is the prin- 
cipal who can either allow it or not allow those programs to happen 
and principal preparation programs now. Again, we are having a 
big turnover of principals. We have an opportunity to prepare ad- 
ministrators who understand what good instruction is, who can 
support professional learning communities of teachers. So hopefully 
in the future we can — our professional development will look dif- 
ferently. We will be able to ratchet it up to a different level. 

Chairman Baird. As the son of a principal who talked a great 
deal about the challenge of aspiring to be the academic leader of 
the institution but being often burdened with the budgeter, discipli- 
narian, police, liaison, et cetera, I think you are right. The aca- 
demic leadership provided by the principal is absolutely critical, 
and I admire the notion that we are going to graduate people who 
actually know what they are doing when they graduate. It really 
is well put. 

Dr. Nelson. Yes, it is not so much the teachers in the field now 
don’t know what they are doing. It is a different world. We have 
learned a lot in the last 20 years, and we are hopefully going to 
get that into the system. 

Chairman Baird. Dr. Weiss, you had some comments? 

Dr. Weiss. Yes, I want to build on what Pinky just said. It was 
a while back, a number of years ago, said that we are putting 
teachers out in immediate need of a 50,000-mile tune-up and that 
unless we improve pre-service education, we will always be at the 
point of remediation rather than continuing education. And we 
need teachers — like all professionals, we need continuing edu- 
cation. One of the root causes of the mess we are in I believe is 
that teachers are asked to try and address far too much content. 
As a result, our preparation can’t be as focused as it needs to be, 
our professional development can’t be as focused as it needs to be. 
It is scattered resources. 

A second comment I wanted to make, for reasons I have never 
understood, we are doing better in developing systems to support 
teachers and principals in mathematics education than we are in 
science education. The notion of professional development mate- 
rials, models and materials that have been carefully crafted, evalu- 
ated, and improved, scaffolds the efforts of lots of people and lowers 
the capacity that is needed to do these well in the field. Building 
on what Michael said, we need efforts, direct efforts, and I don’t be- 
lieve this is a federal R&D mission agency responsibility, but direct 
efforts to build the capacity so that we people our school systems 
with people who are ready to take advantage of the knowledge and 
tools that are out there. I could go on longer but I will stop. 

Chairman Baird. Very well put. I am going to yield some time 
for Dr. Ehlers. He has a follow-up question. 

Mr. Ehlers. Thank you for yielding. I totally agree with you on 
that issue, and in terms of why science takes second seat to math 
I think is pretty evident. Everyone thinks that reading and math 
is something that everyone should understand. I find a lot of people 
who still think that science is only for someone who is going to be 
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a scientist or an engineer. And even if they are teachers they don’t 
regard it as highly important. That is changing. But the way I got 
into science education was just when I was a young professor and 
I was very concerned about scientific illiteracy and I asked myself, 
what can I do as one person? And I decided to set up a special 
course to teach future teachers both science and how to teach 
science. And I thought that was my role in life until it inadvert- 
ently brought me here. 

It is absolutely crucial. And I continue to speak constantly to uni- 
versity presidents, deans, about the importance of this and above 
all something I learned the hard way, that you have to get the 
schools of education to work with the academic departments. Right 
now almost every campus I visit, it is not that they dislike each 
other, they disdain each other. And the academic folks think the 
people in the Department of Education are all aflutter about edu- 
cation, psychology, and theories of education and they don’t know 
how to teach themselves. And people in the education department 
think about scientists as up in the lofty skies. They don’t know — 
have the slightest ideas about what it is like to teach in elementary 
school. They just don’t talk to each other. I found I had to teach 
myself the lingo of the educators, studied educational psychology on 
my own so that I could communicate with them; and once that hap- 
pened, we had a very good relationship and actually worked to- 
gether. 

But it is tough. The easiest thing to do, easier than professional 
development, is to train the teachers right in the first place. It is 
going to take a lot of work on a national scale to make that hap- 
pen. It is happening, you are doing it, Dr. Nelson, Arizona State 
University has done it quite well and Kansas started some good 
programs, Western Michigan University has. So it is coming. It is 
spreading across the country, but it is still not highly regarded in 
the academic world and it should be. Thank you. 

Chairman Baird. Thank you for those insights, Dr. Ehlers. Mr. 
Carnahan, five minutes. 

Mr. Carnahan. Thank you, Mr. Chairman, and welcome to the 
panel. I apologize for getting here late but I did want to jump into 
these questions here. In particular I want to talk about any suc- 
cesses that you have seen or ideas you can share with us about 
how we can do a better job with partnerships between the front- 
line teachers and the private sector and other science resources. I 
mean, my hometown is St. Louis as we have a wealth of higher in- 
stitution entities there, higher education facilities there, private- 
sector entitles there like Monsanto, Boeing, non-profits like Mis- 
souri Botanical Gardens and the Dane Forest Plant Science Center. 
So we have got this wealth of science and engineering there in the 
community, yet we still seem to have the difficulty getting some of 
that translated into the classroom. And so I guess, kind of a two- 
part question, how can we do a better job partnering with those 
kind of resources to supplement what we do in the classroom, and 
the second part of that is with regard to streamlining our teacher 
certification process for some of those retirees that maybe have had 
full careers in one of those institutions, how they could become 
qualified to teach and be a part of really beefing up what we do 
on the front line of our schools. 
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And I would start with Ms. Froschauer. 

Ms. Froschauer. Thank you, Congressman Carnahan. It is good 
to see you again. 

Mr. Carnahan. Good to see you. 

Ms. Froschauer. As you know, we recently were in your 
town 

Mr. Carnahan. Yes, indeed. 

Ms. Froschauer. — for our major and national conference. And 
we had the ability to enjoy all of those wonderful science resources 
you just mentioned. We had about 10,000 science teachers there a 
couple months ago, so it is good to see you. 

I do want to say something about the cooperation and working 
with agencies. I know that Dr. Nelson had mentioned things like 
posters and things of that sort that have come out of NASA and 
other agencies are not highly utilized by teachers. But there are 
many partnerships and many collaboratives that have brought 
about materials and opportunities that have been very beneficial to 
science teachers. But they must be done carefully, they must be 
well-thought-out. They must be things that are actually going to be 
utilized by the teachers or beneficial to the teachers and translate 
into student learning. NSTA partnered with NASA, and in that 
partnership we considered what science teachers need. Now, many 
people have already alluded to the fact that science teachers are 
sometimes are uncomfortable with the content, especially when you 
get into elementary grades, middle schools many times, and even 
a high-school teacher who may be teaching outside of their dis- 
cipline. We know that that content is necessary. We also know that 
NASA has the capacity of providing scientists and the people who 
can actually contribute the content knowledge. And so we utilized 
that content knowledge, that expertise with NASA, pulled it to- 
gether into something that would provide teachers with contact in- 
formation. 

We also know that teachers are busy, and so they want to be 
able to gain that content information kind of in an on-demand sort 
of basis and so we have provided that actually on the web so that 
they can self-instruct, they can also self-evaluate and actually can 
develop that content knowledge, and we do that through something 
called science objects. And they are available free. 

Mr. Carnahan. Excuse me, is that a specific website, Science Ob- 
jects? 

Ms. Froschauer. You can go into the National Science Teachers 
Association website. You will find science objects, and that is actu- 
ally at the front of the wall so that everyone can have access to 
them. And we also partnered with NOAA and FDA on pulling to- 
gether those sorts of things. 

And then during the St. Louis conference for instance we had 
many sessions that were delivered by NASA scientists, by NOAA 
scientists, and also they had space on our exhibit floor where they 
could disseminate materials and share information with teachers. 
So there are some very strong partnerships when you consider 
what the needs of the teachers are and what the resources are of 
the agency and how you could pull those together to benefit teach- 
ers. 
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Mr. Carnahan. And so you think we can do the same type of 
things with local resources as well? 

Ms. Froschauer. You certainly can. And I think you probably al- 
ready realize that there’s a great deal of effort right now at the 
state level to have coordinating bodies established similar to what 
is being recommended through the NSB Council and also through 
the current ACC recommendations, that there would be a coordi- 
nating body that would look at all of the efforts that are taking 
place at a state level and that came out of the governor’s report. 

Mr. Carnahan. I guess the second part of my question with re- 
gard to potentially being able to certify some of those retired ex- 
perts, engineers, scientists that are in communities like that. Has 
there been any work going on that has helped to facilitate that that 
you’re aware of? 

Ms. Froschauer. Well, every state seems to have their way of 
providing for that kind of certification. There are many opportuni- 
ties for people to change a career path and go into science edu- 
cation, and it doesn’t always demand that they go back to college 
for four years to get an education degree. It is actually an alternate 
path to science teaching. And every state seems to have a different 
way of going about that. That is something else that could really 
use some coordination because obviously in some situations it is 
more successful than in other situations. 

Mr. Carnahan. I guess I will ask this of anyone on the panel. 
Are you aware of any — certainly the state certification process var- 
ies widely, but are there any sort of best practices out there that, 
you know, a certain state is doing that we might try to copy or 
other states could look to as a leader in that area? 

Mr. Lach. One of the problems we have in Chicago is the way 
we certify teachers is complicated, it is bureaucratic, and it is not 
the same as our neighboring states. So I think there is possibility 
for ways to streamline that process and perhaps find some com- 
monalities about what a high-school biology teacher or middle- 
grade science teacher ought to know and be able to move things 
ahead. 

Let me speak a moment about your previous question which I 
think is a fascinating one, sort of how would you think this through 
in Missouri or in St. Louis. I think it begins by having a clear plan, 
probably at the state or the LEA level. That takes a fair amount 
of work to put that together and a clear theory of action so you 
would understand why the different aspects of the plan might re- 
sult in a boost in student achievement. If you have a plan you can 
then position the various partners to take on different roles of that. 

In Chicago, for instance, a big part of our plan is a core cur- 
riculum adoption and implementation. UFC does one of our math 
programs, UIC does another, Loyola does our science programs. 
They have become implementation centers within the district to 
move that along. 

The second part of our plan is to increase the content knowledge 
of our teachers by taking university courses. We have 10 local uni- 
versities that we provide tuition stipends for so teachers can go 
back to school and earn the state endorsement in mathematics or 
science. In our plan we also have a place for this inspirational, this 
wow factor. We use the tremendous museums that are in Chicago, 
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the Museum of Science and Industry, the Field Museum, Adler 
Planetarium, others. We the Argonne and Fermilab. We use a lot 
of the community resources to do that inspiration sort of work. 

It has only been in the past year or two when we have had this 
plan and — we have kept with this plan for four years but it has 
only been in the past year or two where I have had enough political 
clout to be able to tell someone, you know, those posters that you 
are providing and those one-shot lesson plans, I don’t think we 
really need that. We have got a plan that is working, we are stick- 
ing to that. But it is very, very difficult to do. We don’t have a lot 
of resources in Chicago. It is pretty unconscionable to turn them 
away. Having a plan, having lots of partners invested in that plan 
enables us to bring that coherence and that support. 

And the last part of that that I also think that you might be able 
to help with is we need tools to sustain those sorts of plans and 
that kind of work. That means high level of capacity in districts 
and in states for leadership in math and science education and the 
appropriate amount of political cover so those people can make the 
kind of decisions that are going to help kids in the long run. It is 
very, very difficult work. It is really difficult to scale, and we need 
lots of resources and help from you all if we are going to give our 
kids what they really deserve. 

Chairman Baird. Thank you. I am going to ask Dr. Weiss a 
somewhat different question but appreciate the tremendous in- 
sights, and then Dr. Ehlers, and then we will open it up for final 
remarks if anybody has any burning issues. So you will get each — 
if you have got something you just haven’t had a chance to say, we 
will get to that, too. 

Just one fairly brief question. Dr. Weiss, questions can be brief, 
the answers often are much longer but the ACC report talked 
about evaluation and the need for a control group design, outcome 
studies. I wonder if you have any — personally I have some mixed 
feelings about that with the internal and external validity issues 
but I would welcome your insights as an expert in this field. 

Dr. Weiss. There is no question that we need to be doing a more 
rigorous job of evaluating programs. I think that the report ignores 
or underplays some really key issues, one I mentioned earlier in 
terms of having measures of outcomes of interest. Another would 
be, when people talk about randomized control trials, they tend, al- 
though the report did talk about this, they tend to not realize that 
we need multiple such studies. A program that works in a rich, 
suburban district, you can’t just now say it is going to work every- 
where else, and that has been a problem. 

But the other is the realities of school districts. If we were going 
to try and evaluate let us say a good set of instructional materials, 
you would want to have some teachers using it and some other 
teachers not and you would want to do that for a long enough pe- 
riod of time so you could look at the differences. Teachers need op- 
portunities to learn how to use the materials, practice, get feed- 
back. School districts are not willing to do that. They can’t have 
two simultaneous programs going on and create whole systems 
around it, and in these days with the pressures, all of the pres- 
sures on school districts, we are in this not-in-my-backyard. Every- 
body wants research-based programs but nobody wants to partici- 



53 


pate in the research that will generate that knowledge. So I think 
there is a whole host of issues, on theoretical grounds you cannot 
argue. On practical grounds you can. 

Chairman Baird. Dr. Ehlers had a question and then we will 
conclude in a second. 

Mr. Ehlers. Just something I wanted to get on the record. If 
there is a lack — first of all, I wanted to ask, if there is a lack of 
coordination among federal agencies regarding STEM education 
programs, how would you recommend solving the problem? How 
should we coordinate it and who should be tasked with the coordi- 
nation. Any ideas on that? 

Chairman Baird. I nominate Mr. Lach. 

Mr. Lach. I was afraid of that. The need is clearly there. I think 
what we found in Illinois is that different districts have different 
needs so I suspect there would have to be some sort of statewide 
localization to address some of those sorts of things. I think there 
also needs to be — I would encourage that the work of the coordi- 
nating — this coordinating body, I am not sure who it should be. I 
think an important role might be to provide really formative work 
to districts and states to help them get better. I get a report card 
every year when I see my test scores, but changing education sys- 
tems, particular at scale, is a very complicated, murky business; 
and if I had a well-regarded report that told me based on the in- 
puts I am putting in as well as the outputs, what I need to do a 
little more of, what I need to do a little less of, that would help 
me organize resources, redeploy my people in a way that would 
help sustain that sort of program. In Chicago, we would love that 
to be really transparent. We don’t have anything to hide. We think 
we are doing pretty well, but we know we have a long way to go. 
So some sort of public report cards about both the inputs and our 
outputs which I think are already public I think would help us im- 
prove and would help make sure that mathematics and science 
were really on the agenda of everybody as we move things forward. 

Mr. Ehlers. Does anyone else want to respond? Dr. Weiss. 

Dr. Weiss. I think we need research that is focused on a smaller 
number of problems of practice. Right now the research enterprise, 
it is pretty wide open and so it doesn’t tend — we don’t have mecha- 
nisms right now for accumulating knowledge on key problems. Mr. 
Carnahan talked about best practice. We don’t really have mecha- 
nisms now for knowing. I thought about the question of where are 
the good, lateral entry programs, and I suspect I know the param- 
eters of what an effective approach would be but I don’t have any 
data and I don’t know that anybody has pulled that together. So 
we need mechanisms for accumulating knowledge. But we also 
need incentives for using that knowledge. I was struck by the com- 
ment in the ACC report that they saw no examples of a federal 
agency building on the knowledge or models of another federal 
agency. So I think it is capacity issues but it is also incentives 
issues. 

Mr. Ehlers. Who do you think would be the best agency or per- 
son to coordinate all that? 

Dr. Weiss. Every candidate that comes to mind has baggage, so 
nothing is quite coming to mind right now. 
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Chairman Baird. The distinguished Ranking Member would be 
an outstanding choice. 

Dr. Nelson. Can I make one quick comment on that? I remem- 
ber back it must have been in the late ’80s there was a group 
called the Fix-it Committee, the federal Coordinating Committee on 
Science and Technology; and it started out with great promise I 
think because it was supported at a very high level from many of 
the different agencies in the — federal agencies, and then kind of 
petered out as the level of participation filtered down deeper into 
the organizations. And it seems like a committee like that that 
could stay at a very high level and provide a focus and some level 
of coordination supported somehow by Congress or someway. That 
is what you guys are good at, right? And would be able to maintain 
that focus rather than expecting to — you know, if you could keep 
a committee like that going for long enough, you might be able to 
bring some kind of a coherence and focus to the program. 

Mr. Ehlers. Any other comments? Thank you very much. I yield 
back. 

Chairman Baird. We have time in an outstanding group like 
this. I would open it up if anybody among you has any critical in- 
sights or issues we haven’t had a chance to address throughout 
questioning that you feel would be a shortcoming if we didn’t raise, 
let me give you an opportunity now. You don’t have to, but if you 
feel like there is something that really we should put down. 

Dr. Nelson. I have a brief comment about research that I would 
like to make. There is no doubt that if you have a controlled lab- 
oratory condition or you can do controlled experiments like double- 
blind drug testing, things that that is a great way to learn things, 
but as a research astronomer, I can tell you somehow astronomers 
learn things and we can’t control anything. And so it is possible by 
posing questions well and by doing a carefully designed observation 
and analysis, it is important to learn things about complex sys- 
tems. To make a point that we can learn important things about 
the education system by carefully — by designing and carrying out 
careful observations and analysis of existing systems, then we can 
apply so not all research has to be — has to fit into this narrow 

Chairman Baird. I think that is a great example and maybe a 
way of an appropriate analogy but would we have seen some star 
programs ourselves today. We didn’t necessarily need a double- 
blind study to observe the effectiveness of those stars. 

Dr. Nelson. Certainly they are important. 

Chairman Baird. Let me suggest — we were kicking around as 
this hearing moved forward, I think there is so much information 
that has been useful today that what we are going to do, we will 
have a — we have already scheduled a hearing with the mission 
agencies themselves for June 6th. We will provide the heads of 
those agencies with the testimony provided by you folks today be- 
cause I think it is so outstanding I would hope they would incor- 
porate the insights in their own work on the educational front. We 
are also exploring the idea of possibly posting this hearing, and Ms. 
Froschauer, we might like to talk to you about posting this hearing 
in some way where it would be accessible to your members. They 
could share some of the models they have heard about how, for ex- 
ample, the science museums are working, how the graduate edu- 
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cation programs are working, how Chicago is making their 
changes, how they might design their research interventions, and 
then offer comments analogous to the approach Dr. Ehlers was 
talking about so that we could actually get further impact of your 
profound insights but also maybe some additional people pitching 
in and create a little bit of a dialogue, and that way we have this 
tool. So we will, if we may, talk to you about if there is a forum 
to do that, a bulletin board kind of model or something like that. 
Without any final comments on your part, Dr. Ehlers? 

Mr. Ehlers. Well, put it on YouTube. 

Chairman Baird. He said we could put it on You Tube and the 
students — well, we do have this extraordinary tool right now. I was 
saying to Dr. Ehlers when I was teaching psychology I came up 
with a pretty neat way to teach about standard deviations, little 
lab experiments you could do, published it in the Journal of Teach- 
ing of Psych if I remember and gosh, that is a rather inefficient 
way. It was a nifty little thing and it worked I think and hopefully 
some people adopted it and used it. But with the Internet you can 
get those things out there so well if there is a coherent way of 
using it. So we are going to try to model that through this hearing 
itself. 

With that, I want to thank our witnesses and our guests today 
and the Ranking Member and the staff for all their work and look 
forward to continue dialogue; and I am confident this will actually 
have some positive impact and appreciate very much your time and 
expertise, and the hearing stands adjourned. Thank you very 
much. 

[Whereupon, at 11:40 a.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Linda K. Froschauer, President, National Science Teachers Association 

Question submitted by Representative Eddie Bernice Johnson 

Ql. The witnesses provided a range of views on how R&D mission agencies can best 
contribute to STEM education. To what extent is there a consensus among pan- 
elists about what the agencies can do well and what they are less well suited 
to do? 

Al. Federal agencies should focus primarily on improving student achievement, 
teacher quality, and student engagement. Specifically, the agencies are well suited 
to: 

Make scientific laboratory experiences and equipment available to teachers and 
students (Dept, of Energy) 

Encourage scientists to work with teachers to further their content knowledge 
(Dept, of Energy) 

Work with universities, the science community and others on model, research- 
based programs to increase teacher effectiveness that can be replicated (Math and 
Science Partnerships) 

Develop and implement proven, effective, research based instructional materials 
and methods (NSF Instructional Materials) 

Recruit and retain teachers with majors in STEM fields (NSF, Noyce) 

Increase the content knowledge of in service teachers with long-term, quality pro- 
fessional develop (NSF, Math and Science Partnerships) 

Support activities that encourage under-represented groups to enter and remain 
in the STEM fields (NSF) 

Increase the awareness and interest in STEM through informal science activities 
(NSF) 

Conduct and disseminate key research in all STEM education fields (NSF) 

Provide undergraduate grants and loan forgiveness for STEM teachers (Dept, of 
Education) 

The agencies are less well suited to develop curriculum or provide professional de- 
velopment. If this is done, it should be accomplished through strong partnerships. 
Although teachers enjoy receiving posters and activities for use in their classrooms, 
these materials do not support the development of conceptual understanding 
through a strong curriculum. In addition, many of the federal agency programs 
sponsor smaller, issue based contests and training for educators. These programs 
simply do not reach the number of teachers necessary to have an impact on student 
achievement. 
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Answers to Post-Hearing Questions 

Responses by Michael C. Lack, Director of Mathematics and Science, Chicago Public 

Schools 

Questions submitted by Representative Vernon J. Ehlers 

Ql. You mentioned that the Federal Government could provide some critical “polit- 
ical cover” to decision-makers at local levels to help advance STEM education. 

Do you have any ideas on the best mechanisms to provide that cover / leadership? 

Al. To best answer this question, let me take a step backwards and articulate some 
of the factors that in my perspective constrain the system for improving K-12 
STEM education. 

• The general public does not understand science or its practice. Sadly, most 
members of the education establishment (school boards, district offices, ad- 
ministrators, state boards of education, principals, etc.) share this lack of un- 
derstanding about STEM issues. 

• The political dynamics in most school systems work against long term solu- 
tions. Given that most school boards are elected to short terms, most super- 
intendents in urban areas last only a few years, it’s difficult to institute 
broad, 5-10 year plans. 

• We know much about large scale change, but have far from a precise under- 
standing about how to move large systems of schools forward. 

These three factors are at the core of our inability to move solutions forward. I 
think the potential exists for the F ederal Government to help alleviate some of these 
issues by establishing: 

Now, imagine if every state superintendent and the board president of the 50 
largest school districts received a detailed “audit” of their mathematics and science 
program once a year. They already receive a performance report — in the form of the 
annual test that the state mandates. Yet test data merely tells states and districts 
what to improve, and doesn’t help them decide how to improve it. The difference 
is critical. 

Most districts know the general direction they must travel, but lack a roadmap 
that shows them how to get there. 

The audit I’m imagining would be conducted by a well-respected, external organi- 
zation. It would review the output data, certainly, but also the inputs — the systems 
and structures the state or district had in place that resulted in the outputs. For 
instance, there’s pretty compelling data that shows a “managed curriculum” ap- 
proach — a standards set of instructional materials for mathematics and science, cou- 
pled with workshop professional development and in school coaching support — is 
necessary for significant improvement. Another example is local school leadership 
capacity, as principal understanding of mathematics and science improvement strat- 
egies and processes also is a major contributor to student achievement. This set of 
standards or conditions for state/district improvement would be challenging to de- 
velop, but I’d posit that there exists enough consensus within the education commu- 
nity that they could be established. In the course of the audit, a team would visit 
the state/district, spend some time digging into both the achievement data and the 
programmatic data, and then prepare a public report to be shared. Perhaps the 
audit or report uses letter grades or a ranking to indicate the status of key aspects 
of a STEM program, so it is easily understood by the public and news media. 

Why would this help? Such a report would highlight both the summative and 
formative aspects of systemic improvement to all interested parties. As the reports 
were issued each year, trends and tendencies would emerge, enabling state/district 
staff to make adjustments and new decisions in light of their review. If they are 
issued with regular frequency from an external organization, much of the internal 
politics within states and districts would be muted. If a board decided to fire a su- 
perintendent and change strategies, they’d do so knowing there will be a report 
forthcoming that will hold them accountable to established national understandings. 
If a superintendent decided to put a large portion of next year’s budget into a new 
reading program, the shift in resources would likely reflect negatively on their 
STEM education report. A board president could use lukewarm ratings on such a 
report to galvanize support for reform efforts throughout the state or city. 

Who could do this work? That’s a difficult question, and probably the biggest prob- 
lem with this entire idea. I think a research-focused organization like the National 
Academies could lead the development of the standards or conditions for large-scale 
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improvement, probably commissioning some additional research in the process. I 
worry that the National Science Foundation doesn’t have enough reach to other fed- 
eral agencies — NASA, NIH, DOE, etc. -involved in STEM education, and that the 
Department of Education just doesn’t have the respect of the scientific community 
to pull this off. A new not-for-profit entity, funded by foundations and business, 
probably wouldn’t have the credibility either without really strong connections to 
the Federal Government, and I can’t foresee a business model where this work 
would pay for itself in the out years. I’m left thinking that the best (though imper- 
fect) answer would be a new agency under the governance of the National Acad- 
emies — there’s enough clout there to handle the academic needs, and enough impar- 
tiality to preserve the inevitable political fights that will ensue. But that’s a consid- 
erably more activist role for them that would entail considerable rethinking of their 
organization and mission. 


Questions submitted by Representative Eddie Bernice Johnson 

Ql. The witnesses provided a range of views on how R&D mission agencies can best 
contribute to STEM education. To what extent is there a consensus among the 
panelists about what the agencies can do well and what they are less well suited 
to do? 

Al. I believe that the witnesses exhibited a large degree of agreement, particularly 
about the important role that the federal R&D mission agencies have to play in K- 
12 STEM education, the fact that much of the current work merely adds confusion 
and incoherence to an already complicated system, and that partnerships with 
states and district education agencies are essential if the work is going to signifi- 
cantly advance student learning. 
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Answers to Post-Hearing Questions 

Responses by George D. Nelson, Director of Science, Technology, and Mathematics 

Education, 'Western Washington University 

Question submitted by Representative Eddie Bernice Johnson 

Ql. The witnesses provided a range of views on how R&D mission agencies can best 

Contribute to STEM education. To what extent is there a consensus among the 

panelists about what the agencies can do well and what they are less well suited 

to do? 

Al. I think there is considerable consensus among the panelists. I would also like 
to raise one point where there is not disagreement among the panelists, but not con- 
sensus either. 

We agreed on the strengths that the R&D mission agencies could bring to STEM 
education: 

• A high quality science and engineering, workforce engaged in cutting edge re- 
search and technology development. 

• World-class laboratories and facilities. 

• The capacity to engage in long-term projects. 

• Existing partnerships with university and industry researchers focused on 
mission-related research. 

• Charters to disseminate the results of mission research broadly within the 
government, to industry and to the general public. 

We also agreed on general weaknesses in the agencies with respect to STEM edu- 
cation: 

• Lack of knowledge of the K-12 education system, now it is structured and 
regulated. 

• Lack of internal expertise in curriculum development, effective instruction, 
and teacher preparation. 

• Lack of expertise in education research and program evaluation. 

If we accept these ideas, then it makes most sense for the agencies to target their 
efforts on informing the public and inspiring the next generation of STEM workers 
through the informal education community — museums and media. Interactions with 
the formal K-12 education system should involve close partnerships with STEM 
education professionals and education researches/evaluators. 

There is some danger that a group deeply engaged its K-12 STEM reform, like 
our panel, sees the world predominantly through a single lens. It brings to mind 
the old saying, when all you have is a hammer, everything looks like a nail. As the 
only member of the panel with a background as a research scientist as well as K- 
12 STEM education reform, I feel I should comment on the role of the R&D mission 
agencies beyond K-12. I believe that the agencies can contribute more significantly 
to increasing the quantity and quality of the STEM workforce by focusing their ef- 
forts primarily on supporting faculty and students to engage in mission-related re- 
search. 

Given limited funding and the wide range of STEM education activities that can 
contribute to the goals stated in the recent reports Rising Above the Gathering 
Storm, and the Report of the Academic Competitiveness Council, each agency should 
focus their resources where they can have the largest impact. In the case of the 
R&D mission agencies, this is mission-related research as discussed in my formal 
testimony. 
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Answers to Post-Hearing Questions 

Responses by Van R. Reiner, President and CEO, Maryland Science Center, Mary- 
land Academy of Sciences 

Question submitted by Representative Eddie Bernice Johnson 

Ql. The witnesses provided a range of views on how R&D mission agencies can best 
contribute to STEM education. To what extent is there a consensus among the 
panelists about what the agencies can do well and what they are less well suited 
to do? 

Al. The R&D mission agencies should stay away from writing curriculum and con- 
centrate on being a content resource. As such, the agencies should partner with in- 
formal education institutions to help communicate the science behind the research 
and mission. Educating the general public will help the formal education institu- 
tions by having a knowledgeable public to reinforce the students’ experience in the 
classroom. 
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Answers to Post-Hearing Questions 
Responses by Iris R. Weiss, President, Horizon Research, Inc. 

Questions submitted by Chairman Brian Baird 

Ql. The Academic Competitiveness Council recommended in its May 2007 report 
that “Funding for federal STEM education programs designed to improve STEM 
education outcomes should not increase unless a plan for rigorous, independent 
evaluation is in place, appropriate to the types of activities funded.” The report 
later describes “rigorous evaluation” methods in a hierarchy with experimental 
methods, such as Randomized Controlled Trials (RCTs), as being the best meth- 
odology for showing the effectiveness of a program. In cases where an RCT was 
not feasible, the next best methodology would be to use a Well-Matched Compari- 
son Group Study. 

Qla. As an expert in evaluation techniques for STEM education programs, do you 
feel that these evaluation techniques are the most appropriate for determining 
the effectiveness of STEM education programs at the federal R&D mission 
agencies? 

Ala. As I noted in my testimony to the Subcommittee, there is no question that 
STEM education programs, including those of the federal R&D mission agencies, 
should have more rigorous evaluations. Randomized control trials (RCTs) are an ex- 
cellent evaluation strategy when feasible. However, there are a number of con- 
straints to the feasibility of this approach that were, in my view, not adequately ad- 
dressed in the May 2007 report of the American Competitiveness Council (ACC). 
First, random assignment to experimental and control groups, or quasi-experimental 
designs with well-matched comparison groups for that matter, are of little use un- 
less there are appropriate instruments available to measure outcomes of interest. 
For example, in Goal 2, a suggested metric for teacher quality is percentage of 
teachers demonstrating increased competency in a given area. However, there are 
only a handful of measures of teacher competency in specific science content areas, 
and even fewer in areas closely aligned with the missions of the federal R&D agen- 
cies, so in many cases even the most rigorous experimental designs would not pro- 
vide useful information about the impact of a program on teacher knowledge. (I 
strongly recommend that the federal R&D mission agencies identify a set of key 
goals, and that NSF be asked to coordinate the development of appropriate meas- 
ures of those goals so the agencies will be in a better position to judge the effective- 
ness of their programs in the future.) 

Second, in my view the ACC report underestimates the difficulties and costs of 
mounting randomized control trials, especially if the intervention is provided over 
an extended duration or if the impact needs to be studied over a substantial period 
of time. To avoid spill-over from the experimental to the control group, teachers 
would need to refrain from collaborating with one another, which is directly counter 
to current recommendations for establishing learning communities within schools 
and districts. Alternatively, the school or district rather than individual teachers 
could be the unit of random assignment, but that typically results in a larger and 
much more costly experiment. 

Third, discussions of RCTs as the gold standard for evaluation tend to focus pri- 
marily on judging impact, downplaying the need to understand not just if a program 
is effective, but also how it works and under what conditions. A truly rigorous pro- 
gram evaluation would include exploring the nature and quality of implementation, 
as well as both proximal and distal outcomes, in a variety of contexts; one or two 
RCTs or quasi-experimental studies focusing only on the ultimate impact of a pro- 
gram, e.g., student achievement, leave many important questions unanswered. 

Qlb. What advice would you give the federal R&D agencies for developing evalua- 
tion methods to help them judge which programs are effective and whether pro- 
posed programs will be successful? 

Alb. There are many, many needs in science education, and many possible ap- 
proaches to improving the situation. But given the magnitude of the problems and 
the scarcity of resources for addressing them, it is important that funding be limited 
to programs that will make a substantial difference. In my view, the first step is 
for each proposed program — and continuations/modifications of existing programs — 
to specify its goals, in effect describing what would count as success. Agencies would 
be held accountable for achieving their specified goals, so there would be an incen- 
tive to identify goals that are both realistic and measurable. 



64 


Once the goals of a proposed program are specified, the agency should describe 
the theory of action underlying the program design — how the planned activities are 
expected to lead to the desired outcomes — citing available evidence that supports 
particular elements of the program design. For example, teacher professional devel- 
opment programs do not directly affect student attitudes, aspirations, or achieve- 
ment. If a professional development program’s goals include impact on students, 
then a case needs to be made how the program activities will result in those im- 
pacts. A sample theory of action might be: Professional development activities will 
focus on concepts A, B, and C (a small number of important concepts addressed in 
national standards for the target grade range that are relevant to the mission of 
the agency), providing opportunities for teachers to explore these science concepts, 
to learn about applications of these concepts in the work of the agency, and to con- 
sider how their instructional materials can best be used to develop student under- 
standing of these concepts. Teachers will gain a better understanding of the content, 
applications of the content, and use of their instructional materials, and be able to 
use their enhanced knowledge in their instruction. Improved instruction will enable 
students to see the relevance of the targeted content, motivate them to learn these 
concepts, and lead to improved student knowledge of those science ideas. 

Is the program well designed to achieve its goals? Prior to expending resources 
on implementation, it would be important to have the program design, theory of ac- 
tion, and evidence presented in support of the design critiqued by a small number 
of external experts, similar to the process used by the National Research Council 
for review of committee reports. Program designers’ responses to these critiques, in- 
dicating how reviewers’ concerns will be addressed, would be reviewed by program 
managers with input from another external expert acting as “monitor”; only those 
programs that appear to have responded adequately to reviewers’ concerns should 
be allowed to go forward. 

The next step that should take place before program implementation is the design 
of an evaluation to assess both the quality of implementation and its impact, using 
the program’s theory of action as a guide. Suggested connections, for example be- 
tween program activities and teacher content knowledge, or between teacher content 
knowledge and classroom practice, can be considered as hypotheses to be tested, and 
the evaluation plan should describe how that testing will take place. (It is important 
to note that an evaluation plan needs to specify both the measures to be used and 
the research design that will enable the evaluation to make the case that any gains 
that are found are due to the program and not extraneous influences.) Only pro- 
grams that can be adequately evaluated should be considered for funding. It makes 
little sense to devote resources to an endeavor if you won’t be able to determine if 
it is successful. 

If a program has a promising design, and can be evaluated, it should be piloted 
on a small scale to see if it can be implemented with quality and to check to see 
if the proposed measures are sensitive to the interventions. If not, there may be 
problems with the program and/or there may be problems with the instruments, the 
program should not be scaled up until these problems are identified and resolved. 

Only when a program has cleared these hurdles — a program design that seems 
promising based on prior evidence, can be implemented with quality, and can be 
adequately evaluated — does it merit broader implementation, and then it needs to 
be carefully evaluated to see if the program is in fact achieving its goals, and under 
what conditions. 


Questions submitted by Representative Eddie Bernice Johnson 

Ql. The witnesses provided a range of views on how R&D mission agencies can best 
contribute to STEM education. To what extent is there a consensus among the 
panelists about what the agencies can do well and what they are less well suited 
to do? 

Al. I believe there was considerable consensus on a number of issues: 

• The federal R&D mission agencies have an important role to play in the infor- 
mal education arena — including museums and science centers — in increasing 
students’ interest in science, encouraging them to consider science careers, 
and enhancing science literacy of the general public. 

• It is typically not helpful for the agencies to develop posters and activities for 
classroom use. Even if these materials are very well aligned with national 
and state standards, and many are not, they add noise to an already noisy 
system, making it less likely that students will encounter a coherent science 
curriculum. 
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• If the agencies are going to be involved in teacher professional development, 
they need to partner with groups that have expertise that goes beyond con- 
tent knowledge to include developing teacher pedagogical content knowledge 
and skill in applying what they learn to the classroom using the instructional 
materials selected by their districts. 

Q2. How would these programs need to be designed so that they can achieve class- 
room effectiveness ? 

A2. The teacher professional development programs carried out by the federal R&D 
mission agencies have the potential to enhance teacher content knowledge. (It is dif- 
ficult to know the extent to which they have actually done so, as in the absence of 
appropriate measures, evaluations of these and many other programs have had to 
rely on teacher self-report to assess impact.) Getting transfer to classroom practice 
has proven to be more difficult. The evidence we have available suggests that to fos- 
ter improved classroom instruction, professional development needs to be sustained 
over time, include a focus on pedagogical content knowledge, such as common stu- 
dent misconceptions in a specific content area, how to assess student understanding 
of the target concepts, and how to use that information to improve teaching and 
learning. In my view, the agencies are not well-positioned to provide sustained, pro- 
fessional development focused on helping teachers not only learn content, but also 
apply that content to their instruction, but if they are going to do so, it would be 
helpful if the agencies partnered with groups that have the necessary expertise. 

In contrast, I believe the agencies could make an important contribution by pro- 
viding opportunities for prospective secondary science teachers to have an authentic 
research experience as part of their pre-service, or lateral entry, preparation, a 
much less expensive and potentially more scalable strategy than current programs 
targeted at that goal. Each of the agencies could develop research modules relevant 
to their mission and aligned with national standards that could be incorporated into 
existing college/university courses, with college/university faculty and/or agency sci- 
entists serving as mentors depending on the context. 
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Evaluation Background and Methodology 
Introduction 

The Maryland Science Center contracted with RMC Research Corporation in June 2004 
to conduct a front-end evaluation of Science on a Sphere, a new spherical film 
technology developed by the National Oceanic and Atmospheric Administration. The 
Maryland Science Center mounted Science on a Sphere for a two-week exploratory 
period, between June 15-30, 2004. The front-end evaluation aimed at assessing the 
educational strengths and potential of Science on a Sphere as exhibit technology in the 
museum setting. 

Science on a Sphere is a six-foot diameter sphere that is suspended from the ceiling (or a 
specially designed frame). Driven by a bank of computers, four video projectors arranged 
around the Sphere display spherical images created from a potentially limitless number of 
data sets. Data sets used in the Maryland Science Center presentations included images of 
Earth and other bodies in our solar system. Some of the Earth projections include infrared 
satellite data depicting the development of storm systems as they move across the globe, 
sea surface temperatures depicting el nifio and la nina events, an animation depicting 600 
billion years of continental drift, and images of Earth’s topography and bathymetry. 
Additional data sets depicted the surface of the Sun, Moon, Mars, and other bodies in our 
solar system. 

Evaluation Questions 

The evaluation plan developed by RMC was designed to address the cognitive and 
affective aspects of the Science on a Sphere for learning experiences, and was based on 
consideration of three aspects of the exhibit: 

• visualization of data on the sphere 

• interpretive program scripts 

• sphere placement in the museum 

Evaluation questions were developed for each of these areas. 

The Visualization of Data on the Sphere 

• What are visitors’ first impressions of the Sphere? Do visitors understand the 
purpose and content of the Sphere? Do the visualizations convey the intended 
meanings? What kinds of supporting information are needed to make it both 
comprehensible and engaging? 

• How do visitors respond to the visual presentation of data on the globe? What 
kinds of images are most provocative, memorable and engaging? 


RMC Research Corporation: Science on a Sphere Front-End Evaluation DRAFT 
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Interpretative Program Scripts 

• What kinds of information and stories are most suitable for presentation in this 
format? What are its strengths and weaknesses in comparison to the presentation 
of information on a flat screen, in written text or via other media? 

• What specific visual and narrative techniques are most effective in conveying both 
conceptual and factual content? How can complex ideas such as different time 
scales be best presented? 

• How can the Sphere and accompanying scripts be used most effectively for 
depicting global systems and interconnectivity? For instance, how can information 
about the global nature of weather patterns be best presented given the visual and 
auditory aspects of the program? 

• How do the visual and audio components of the program interact, and what are the 
best techniques for ensuring that the audio text supports and enhances the visual, 
and minimizing the degree to which it is inhibitive or over-prescribed? 

Sphere Placement 

• What are visitors’ behavioral patterns of interaction with the Sphere, i.e. do they 
walk around it? Do they settle on a particular position or angle? 

• What effect does the placement in the museum, including such factors as the size 
of the room, amount of traffic and noise level, on visitors’ experiences? 

• How might auxiliary information be presented in association with the Sphere? 

Interview protocols were developed out of these broad evaluation questions, tested and 
refined as described below. During this process, it was decided that the first of the 
evaluation questions, that about the Visualization of Data on the Sphere, was the most 
important for this pilot exhibition. 

Methodology 

The evaluation plan developed for Science on a Sphere included the collection of 
observational and interview data during the two week stay of the Sphere. Data were 
collected in two phases, as follows: 

Phase One (June 1 5-16) - During the first two days of the exhibit, an RMC staff member 
collected observational data, and conducted oral interviews using draft interview 
protocols developed prior to viewing the presentations. These interviews were intended 
as an opportunity to test the protocols, but are valuable as the only data collected of 
school group visits due to the timing of the exhibit. Interviews were conducted by an 
RMC staff member with the assistance of MSC and visiting science center personnel. 
Small groups (3 to 6 individuals) were interviewed, including student groups, their 
teachers, and the general public. 

Phase Two (June 18-30) - Survey instruments were revised based on considerations 
described below and used for the remainder of the exhibit period. Two versions were 
again created to allow a wider range of questions to be asked. At the conclusion of each 
of the approximately eight presentations a day, visitors were asked by the presenter to 
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complete written surveys which had been placed on the projector stands around the 
Sphere on clipboards. 

Interview protocols were revised based on the actualities of Sphere presentations, 
preliminary data, feedback from project principals, and availability of MSC staff for data 
collection, as follows. Contrary to expectations prior to viewing NOAA presentations, 
visitors were expected to move around the Sphere, sometimes as many as six times 
during a single presentation to see particular images on one part of the Sphere. As a 
result, issues originally built into the survey instruments were no longer meaningful. 

In addition, no program scripts had been developed specifically for the exhibit. Rather 
presenters would be using a script developed by NOAA, or speaking extemporaneously 
on topics well-known to the presenter. Thus, project principles requested that the 
evaluation focus on capturing viewer responses to the Sphere as a medium of 
presentation, rather than on the Interpretative Program Scripts. 

Preliminary data and evaluation goals were reviewed at a workshop with Maryland 
Science Center and collaborating museum staff. MSC and visiting museum staff 
members were primarily concerned with how visitors responded to the Sphere, and issues 
such as whether the Sphere could be used without a presenter, and how to think about 
guided presentations in the science center day. 

The goal of the evaluation of this pilot exhibition of Science on a Sphere in the Maryland 
Science Center was to understand viewer responses to the Sphere as an educational 
medium. Towards this end, visitors were asked open-ended questions as a way of 
surfacing the range of viewer responses to the medium. Written surveys that could be 
completed by visitors without the assistance of data collectors were developed for use. 

An RMC staff member returned to the Maryland Science Center for the final two days of 
presentations. During this time, additional observational data was collected on visitor 
behavior, and randomly selected visitors were asked to respond to additional interview 
questions following completion of the written questionnaire. 

School Group Interviews 

Interviews were conducted during Phase One of the evaluation in order to gather initial 
feedback before refining study instruments. Due to the timing of the exhibition, these 
were the only opportunities for feedback from visiting school groups attending Science 
on a Sphere demonstrations. These included two visiting groups of fourth graders and one 
group of eighth graders. At each of these a handful of adult or family visitors also joined 
the presentations. 

Surveys 

A total of 326 surveys were collected. Visitors who completed surveys included 74 
visitors under 18; 28 visitors between 18 and 25; 127 visitors ages 26-49; and 97 visitors 
ages 50 and older. 

Instruments used in both interviews and survey are included in Appendix A. The data 
obtained using both sets of instruments is included in Appendix B. 
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The following discussion draws on survey, observational and interview data collected 
throughout the Sphere exhibition. Responses across survey instruments and method were 
very consistent. 
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Summary of Findings and Recommendations 

Introduction 

The Maryland Science Center hosted hourly free presentations using Science on a Sphere 
during the days of the SOS exhibit. Presentations were announced 1 5 minutes prior to 
the start of the presentation, and varied in length from 20 to 40 minutes. 

The Sphere was located on the third floor, near a group of interactivcs and on the other 
side of the floor from the children’s room. Although the Sphere was not located in a part 
of the science center conducive to visitors spontaneously joining presentations, visitors 
did occasionally do so, or return for a presentation after seeing the Sphere. 

The Sphere was located in a relatively cool, quiet, dark comer of the museum. The 
projector stands acted as unintended benches for visitors to sit on both during 
presentations, and frequently to stop and relax while watching the globe “turn” between 
presentations. A few visitors commented that they would like seats all around, or “better 
seating”; and a few others suggested a darker room. There were also a few comments that 
the PA announcements were distracting. While these are all suggestions for ideal 
conditions for placing the Sphere, nothing about the Sphere’s placement at MSC 
detracted seriously from the experience. 

Presenters 

Five different presenters gave presentations during this time. This included three 
presenters from NOAA and two from the Maryland Science Center. NOAA staff 
presentations followed a shared script, using the same sequence of databases. The 
programs conducted by MSC staff covered areas in the presenter’s expertise. 

The NOAA presentations followed a shared script. These programs included images of 
Earth’s topography and the lights at night, global weather showing the formation of 
storms, several decades of surface water temperatures depicting el ninos and la niixas, 
views of the surface of the Sun, Moon, Mars and Earth (NASA’s “Blue Marble” view of 
Earth as seen from space), and concluded with an animation of 600 billion years of 
continental drift. Some presentations also included a projection of future global warming. 
The NOAA presenters covered largely the same information, though each presenter had a 
distinct style and sense of humor, and varied to the degree to which they shaped their 
presentations in response to particular audiences. These NOAA presentations constituted 
the far majority of presentations conducted over the two-week period. 

Presentations conducted by MSC staff members covered two sets of topics — one focused 
on Earth’s topography and geography and the second on the solar system. The first used 
data sets depicting Earth during the daytime, with the Earth’s topography highlighted on 
one side, and lights at night on the other, thermal data depicting storm formation, 
continental drift, and finally the Blue Marble. In this presentation, the facilitator played 
with the tilt of images of Earth, allowing viewers to get a clear view of both poles. The 
solar system presentation began with the view of Earth from space, then two different 
kinds of imaging of the Sun, the Moon, Mars, Venus, Jupiter, Io and back to Earth. 
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Although evaluation questions did not directly ask about the presentations or presenters, 
there were several comments on both. Visitors appreciated having a live presenter and 
found it helpful for focusing. For instance, one viewer wrote, “The presentation helps 
focus your eye and mind. Even so, it is awesome even if you are just looking at it.” A 
number of visitors commended the presenters for their knowledge of topics covered, use 
of humor, and attentiveness to children. One viewer wrote, “The interpreter was 
excellent! ! He is very knowledgeable and communicates very well. Maybe a waste of 
time with no interpreter.” These two quotes were unique in praising as well the value of 
having a presenter for making sense of the images on the Sphere. 

Level of Engagement 

Structured observation of Sphere programs during the opening and closing days of the 
exhibit, as well as visitor comments suggest visitor engagement during presentations was 
very high. Maryland Science Center staff had originally planned on 20 minute 
presentations, but when the NOAA staff held audiences for 40 minute presentations, that 
became the norm. Presentations were free and maintained a constant number of visitors - 
with the few who left during a presentation replaced by others who joined. The far 
majority of individuals who began a presentation, stayed through to the end. In most 
cases, people were engrossed throughout. In one case, two presenters did back to back 
presentations lasting nearly 2 1/2 hours. Some visitors stayed over an hour, and others 
came and went to catch pieces of more than one presentation. Visitors often asked 
questions, and after almost every presentation, visitors stayed to talk either about the 
technology, the Sphere’s value as an educational tool, or about the content of the 
presentation. 

Parents and teachers frequently commented on how engaged their children were 
throughout the demonstration, which was generally longer than what they had come to 
expect in the Science Center. One parent was impressed that “My kids held on for 30 
minutes.” Another parent described how the Sphere had captured the attention of her 
twelve year-old son. “My son was feeling bored when we were going through some of 
the rest of the museum. He didn't want to come today. He sat through the sphere exhibit 
spellbound.” 

All of the classroom groups observed were highly enthralled throughout the forty-minute 
presentations. In these programs, the facilitator did tailor the content and level of 
interactivity (asking questions) to the school group. In any case, teachers were impressed. 
“They were into it,” noted one teacher, “even some of the more active ones.” 

Very young children were the most difficult audience members to hold. In several cases, 
one parent left with a very young child (varying in ages, but often under 6 years) and 
could be seen at the interactive exhibits just beyond the Sphere area, while other family 
members continued with the presentation. When there were a sizeable number of children 
in the group, some of the presenters kept children engaged by asking questions of the 
audience throughout the program. 

There was little communication between visitors during the presentations. One notable 
exception was observed. A presentation about Earth’s topographical features, with a 
slowly revolving earth, inspired considerable pointing and discussion among visitors. A 
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combination of the slow spin of the globe and the discussion of a familiar topic - Earth’s 
topography - seemed to offer an experience that the viewers could participate in and share 
information among themselves. 

Presenters asked audiences to move around the Sphere several times during the course of 
the program. In most cases, visitors responded positively to this, both in action and in 
their comments. For instance, a group of fourth graders said they, “liked moving around 
the globe, it was fan to look at.” A few were not as comfortable moving about. One 
visitor commented, “Like sphere - didn't like having to move - wanted to sit down.” On 
occasion, visitors simply chose not to move. 

Sometimes a particular image or data set would inspire even those passive audience 
members to rejoin the group at a particular vantage point. Images of the formation of 
Hurricane Isabel, which had hit Maryland the previous summer, was one of these. 
Segments of the continental drift animation were also captivating. For instance, two 
young boys were sitting on the sides, but when the presenter started talking about an 
asteroid striking the earth during the time of the dinosaurs, the boys moved from their 
benches to look at the image of the asteroid’s impact. In another case, a couple who had 
been relaxing on the side got up to see the formation of North America during the plate 
tectonics demonstration. 

Visitor Ratings 

Visitors responded very positively to Science on a Sphere. On a 5-point scale from poor 
to excellent, an overwhelming 98% of visitors across all age groups, rated the sphere as 
either very good (67%) or excellent (3 1 %). Less than 2% (6) gave the Sphere a rating of 
Average and less than 1 % ( 1 ) gave it a rating of Fair. A detailed chart of responses is 
included in Appendix B. 

First Impressions 

Responses to the question “what most impressed you about the Sphere?” offer an 
overview of viewer reactions to the Sphere presentation. Responses to this question 
reflected three main themes heard throughout the evaluation. These include appreciation 
for the Sphere as: 

• an innovative piece of technology, 

« an aesthetic experience, and 

• a versatile educational tool for adults and children. 

Sphere Technology 

Visitors were very impressed with the technology of the Sphere. One exclaimed that “The 
technology is amazing!” and another said, “Magnificent! How do they do it? I spent 
some time trying to figure out how it's done.” Other visitors marveled about “The 
dynamics of setting it up and getting it perfectly placed to effectively show the 
presentation,” and “that two thin wires could hold a sphere that size.” They were 
impressed by “how you can project something onto a sphere,” “the ability to see 
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everything to scale,” and the resolution, clarity, color, and ability to depict historical 
change. 

Others commented on the way in which the projections were made to appear realistic, 
and were impressed by “How it looks like it rotates,” and “How it was exactly the same 
tilt as the earth.” “At first you think the globe is moving. But it's not. It's definitely 
different. . .Good not to have distortion.” One visitor described the Sphere as “modem” 
and “up to date.” Many more responses dealt with the unique ways in which data was 
visualized on the Sphere. These will be addressed in a section below. 

Aesthetic Experience 

While many of the children spoke of the Sphere as “cool,” adult visitors described the 
Sphere as “beautiful” and “riveting.” “The beautiful colors [are] like an abstract painting. 
The globe is gorgeous to look at,” “It's so amazingly beautiful and we are SO small in the 
scheme of things!” “Beauty, reality, content, storytelling by the presenter. Comparisons. 
History. Scale. Simply awed,” wrote one visitor. While another explained that she was 
impressed “Simply by the idea of it - never seen anything like it! I'm not even much of a 
science person - it's just very cool.” One visitor suggested “1 think you should have a 
cocktail partly around it. It's very engaging. It moves.” Others noted that it appeared to 
float in mid-air. 

Even outside of formal presentations, visitors often stopped to rest and watch the “Blue 
Marble,” global weather data, and other images. “It’s also very relaxing to just look at it,” 
commented one such itinerant visitor. A few visitors described the Sphere as “tangible,” 
“Much more comprehensive and tangible,” “It think it becomes very tangible to see the 
actual planets rotating.” This tangible experience of the Sphere was reinforced by the 
numerous passing children who ran up to the Sphere to touch it, to the chagrin of NOAA 
staff. Another visitor described the experience as “intimate.” 

Educational Tool 

Across all age groups visitors mentioned the educational value of Science on a Sphere. 
Among the many visitor comments were that “It makes the images more relevant,” “I 
think it is a great educational tool,” and “It shows dynamism, complexity, weather and 
economic development. Very informative and insightful.” One visitor was impressed by 
“The amount of scientific information that was presented,” and another noted that “It's a 
great visual learning tool.” 

A few of the more extensive comments gathered in interviews suggest the enthusiasm of 
educators and some of the myriad ways they were imagining using the Sphere. A former 
aerospace engineer from Arizona, who will be teaching high school in the coming school 
year, was very excited about the Sphere as a learning tool. He said, “There is so much 
there to learn. So much of presentation you could focus on - Mars, Sun, history of 
Earth. . . He found it helped him to understand his local weather, “Watching weather 
over Arizona vs. other locations, you can see why Arizona is desert.” And said he “would 
love to be able to take a group of 7th and 8th graders. . . This would stimulate kids.” 

A educational grant writer in Texas marveled at her own daughter’s learning, explaining 
that “When he [the presenter] held up the nickel [demonstrating the relative sizes of the 
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Earth and Sun], my daughter said, "Think how small we arc.”” She felt that this lesson in 
perspective would be valuable for the inner-city students she works with in Texas, and 
said “We have inner city kids. The Sphere would help them to see outside of their world - 
help them to see beyond Waco.” She recognized, however, that the program would have 
to be more interactive to hold the attention of the students. She also shared a number of 
ideas for how to use the Sphere for a GPS data project. She marveled, “To hook this up to 
geological and cultural systems data would be cool. With global information systems, 
you can show whole civilizations growing. Could show it on this. Could combine this 
with global positioning system activities. Kids would make a cognitive leap.” 

Another teacher was enthusiastic about using the Sphere for teaching Earth Science as 
well as biology and chemistry. She suggested the integration of children’s animated 
characters as a way of appealing to very young children. “You could have the Magic 
School Bus which just had an episode on Venus, for school presentations to reach the age 
of kids like her 5 year old.” 

One teacher suggested more lessons/more information, including a lecture on all four 
oceans. “Make it more interactive, visual aids of dinosaurs etc for changing earth.” 

Versatility: Visitors were also impressed by the versatility of the Sphere. Some of these 
responses noted the “Richness of the technology and data sets presented,” “The ability to 
show many different topics and time periods,” “The way it could be used to show various 
data sets in their native global format,” and the “Ability to switch views and interactively 
observe events and objects, planets, and their atmosphere.” 

Responses to the question of what impressed visitors most about the Sphere also included 
a number oi mentions of the knowledgeable presenters and appreciation of having a live 
presentation. Many visitors also mentioned particular images, data sets, or facts that they 
were particularly impressed by. These are all addressed in subsequent sections. 

Overall, visitors were very impressed with the Sphere. Many took still or video footage of 
the Sphere. One of the children interviewed liked the sphere so much that he said “I want 
to come for my birthday.” 

Most Memorable Image 

In surveys, visitors were asked to name the image most memorable to them. Virtually 
every data set was mentioned, suggesting that all of the data sets were of interest to at 
least some of the visitors. However, some data sets, particular facts, and images stood out 
for large numbers of visitors. Keep in mind that the various programs included different 
data sets. For instance, with the exception of the presentations exclusively on the solar 
system, the continental drift animation was given considerable time and shown three 
times in most presentations, while images of other bodies in the solar system were 
covered relatively quickly compared to their treatment in the presentation exclusively on 
the solar system. Thus the different numbers of people listing each segment doesn’t 
necessarily mean that some images arc inherently more interesting than others, but may 
reflect the varying amounts of time and attention given to them in different presentations. 

Continental Drift: Almost one third ol the visitors mentioned the animated sequence on 
plate tectonics in response to the question about the most memorable image. As one 
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viewer wrote, “The most memorable image was rather a series of images - the animation 
of plate tectonics and continental drift through millions of years. It gave an easily 
understandable, broad view of continental drift.” The combination of a global view, with 
a historical view - “Pangaea to the present day” - and the visual excitement of seeing 
such dramatic change in the Earth’s plates were all factors in making this memorable. . 
Several viewers mentioned in particular the visualization of time. Some of these 
comments included, “600 million years to present! Time-warp - one of the best learning 
tools I’ve seen,” and “The continental drifts. Just seeing them. This is 3-D (vs. map). 
Books are books. You see it - passage of time.” This sequence was mentioned in 
response to questions about what most impressed visitors, and what they learned, as well 
as the most memorable image. 

Responses throughout the surveys also noted specific occurrences in the plate tectonics 
animation, such as “Indiana moving,” and “India crashing into Asia.” Also of interest to 
many visitors was the discussion of what the Earth looked like in the time of the 
dinosaurs, the story of the comet crash during that time, and changes in Earth’s 
vegetation. Interestingly, none of this information had visual components, yet were 
mentioned throughout the survey responses. 

Global Warming: The global warming data set was the second most frequently 
mentioned in response to the question of most memorable image, but also came up as a 
significant area of learning, and as what some visitors found most impressive. A group of 
students gasped as global warming was projected into the future. One viewer wrote, “ I 
learned how global wanning will affect the earth in 500 years.” Other viewer comments 
suggested concern with the implications of what they saw, “How global warming is a 
reality and will have scary consequences.” And the display left one viewer wondering, “If 
we know about global warming, why isn't anything done about it?” 

Weather: Also frequently mentioned were the images of the weather, including the 
formation of hurricanes and specifically of hurricane Isabel. Several mentioned the image 
of “Storms forming in Africa,” and “The storms coming across the oceans.” Several 
others mentioned being impressed by the ability “To actually see weather systems in real 
time.” Also mentioned were the images of el ninos, and la nifias. A discussion with one 
of the visiting class teachers spoke to the memorable storm images. She said the 
presentation was good for her students. In particular, “The model of Earth and tracking 
stonns is good for them. They’ll remember the eye of the storm. It was a great 
experience.” 

A number of visitors in both interviews and on surveys were taken by a range of images, 
and could not pick out a particular image as most memorable. Some of these comments 
include “All of the images were fascinating,” “All of them were great,” and the “Entire 
presentation was interesting.” 

Earth at Night: Smaller numbers of people mentioned the image of Earth at Night - 
“The way it showed the earth at night and how the light on earth can be seen in space” 
and were impressed by the mapping of human use of electricity. “The presentation was 
very thorough. The human use of electricity was mind blowing.” 

Mars: Several visitors mentioned images of Mars as among the most memorable images. 
Several noted in particular the large canyon and mountain described in presentations and 
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compared with features on Earth. Others mentioned the poisonous gas on Jupiter. The 
Sun, other planets, and moons (Earth’s and Jupiter’s) were also mentioned, though noting 
somewhat less detail. This may be because with the exception of a few presentations 
exclusively on the solar system, these images constituted a much shorter portion of the 
presentation. 

Visualization of Data on the Sphere 

Visitors were asked “What is the point of the Sphere?” This question was meant to get at 
whether visitors understand the purpose of the Sphere and its unique value in visualizing 
data. Visitors unequivocally grasped the unique ways in which the Sphere allows for data 
to be presented, and expressed it through a variety of phrases, capturing the significance 
of the imaging on the Sphere through contrasts with other media. Some offered a broad 
view. For instance, as one visitor stated, “It gives a different perspective that you can't get 
from other media,” Others found more detailed ways of describing the imaging. And 
some found social and political meaning in the global views. 

3-dimensional views: The most common term to describe the uniqueness of the Sphere’s 
imagery was to call it 3-D and to contrast it with “normal” 2-D views of maps and videos. 
“The point of the Sphere is to give the audience a 3-D image of the information being 
relayed. It’s more lifelike,” “It makes the images normally portrayed in a flat, 2-D 
manner moTe tangible, thus making them more "real.” “Sphere approach makes 
presentation much more “actual” - as the components of the solar system really are huge 
steps beyond flat screen approach which we’ve seen all our lives.” 

Realistic or “alive”: As noted earlier, visitors commented on the movement, color, size 
and detail of the imagery, as well as the tilt, rotation and ability to see planetary features 
to scale. Numerous visitors described the Sphere’s imaging as “more realistic” or 
“lifelike.” These included comments such as “It gives a more realistic impression of the 
planets, moon, sun, etc,” “You get to see things like they are,” and “It brings the image to 
life.” It also “reflects how alive our planet is.” "It's more like real life with the sphere.” 
“Shaped like Earth - realistic perspective of relationships.” “It gave us a real feel for the 
shape and events happening on the surface of the planets.” 

Lack of distortion: Visitors spoke of the lack of distortion of the images on the Sphere, 
and contrasted them with a map or video. “On a map or video, you can not walk around 
the image and [on the Sphere] the image isn't distorted. The point of the sphere is to teach 
people about Earth in a new and innovative way.” “One is really seeing the true shapes 
and sizes of images opposed to flattened distorted images found on 2D maps and 
computer screens.” 

Global views: Visitors spoke about seeing the whole planet and the whole picture. Some 
of these comments include: “You can actually see all around,” “You can view the earth in 
its totality,” and “Map doesn't show other ways around. Globe shows it better than a flat 
map.” Many viewers commented on the particular systems for which it was valuable to 
see a global view. These include, “Gives a world view of geological phenomena,” “Gives 
full view of planet weather, plate shift,” and “Very realistic - Can’t possibly present this 
on a flat surface - i.e. night and day.” 
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Global interrelationships: Visitors noted that the Sphere ‘‘shows the relationship of 
events on Earth,” provides an opportunity “To represent various interactions (weather, 
electricity, topography) involved with the planet,” and “To understand how connected the 
earth is; to demonstrate that to people. The sphere allows you to see the interaction of 
weather around the globe,” And “The way weather patterns move from the equator to the 
hemispheres.” 

Several visitors believed this global perspective valuable: “Global perspective is new and 
important in a nationalistic society,” And that it’s “Good to get out of the North America 
- the storm systems begin in Southern Hemisphere. It shows that what happens on other 
continents affects us.” Others noted that it provides context, or the big picture, “Ability to 
visualize effects on a global basis. The big picture.” 

New Perspective: A number of visitors described the Sphere as providing a new 
perspective, suggesting that the images projected on the Sphere challenged how they see 
the world. Several visitors likened this to seeing the view of an astronaut. One student 
noted the value of this view, “being an astronaut allows you to see these things, how it 
rotates, electricity at night - we use a lot of lights, volcanoes in the ocean.” Another 
visitor said, “With the "Earth'' suspended in "air" it felt as if I was in "outer space" 
looking in.” 

In some cases, visitors noted simply the value of having things visualized that hadn’t 
been visualized before. An 8 th grade teacher explained that her students “Knew about 
Earth’s formation billions of years ago, but it was new to see it. Also new was how the 
earth looks at night, and how people use electricity.” She continued that it in fact 
answered questions children had previously asked. She said, “The visual is good for 
them. To see from the astronaut’s view. “What do they [astronauts] see?” That’s a 
question I’ve had in class. . . Seeing nighttime and daytime is interesting. 

In addition, a few visitors noted the value of having time and space visualized, such as 
the visitor who was impressed by, “The perspectives over time and the spatial perspective 
("the big picture").” 

The Local in the Global 

While noting the value of a global perspective, visitors were often drawn to images of 
places with personal relevance. Maryland visitors generally marveled at seeing the path 
of hurricane Isabel, while visitors from Burma to Arizona eagerly looked at weather 
patterns near their homes. Another visitor commented on the highlighting of Indiana in 
the plate tectonics animation, and noted the value of having local places noted. “ [The 
Sphere] is unique for getting a world wide perspective, though it’s always good to point 
out particular places. [In the animation of continental drift] show Maryland not Indiana” 

Other viewers found relevancy and connection based on criteria other than place. A 
woman who commented on why she selected Mars as among her favorite images noted it 
was because Mars had been in the news recently. She explained that she had difficulty 
choosing between Earth and Mars as her favorite image. First she described what 
appealed to her about the image of Earth. “For Earth, you get so caught up in your life - 1 
live in Maryland. When you see the Earth so far away it puts life into perspective. You 
see the whole thing you think about all the wars going on. . . This is 3-D and puts it more 
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visually into perspective.” In contrast, her connection to Mars came through following 
events in the news. “When we look at Mars it's interesting because of the activity going 
on with the Rovers.” These comments suggest that specific details provide a focus, 
relevancy, and personal connection within this global view. 


Would Recommend to Others or Return 

Visitors overwhelmingly said they would recommend the Sphere to others, or return to 
see other presentations. A response from one of the students visiting with his class was, 
“Yes [1 would return]. It was awesome, coolest, the best.” Other visitors said they would 
return because “it gets you up and about to view the various quadrants of the sphere 
rather than just lectures you while you sit in one spot,” and “Absolutely! Very visually 
stimulating from a macro/"God's-eye" view of the world.” And one young woman said 
simply, “It’s bitchin’.” 

Several also gave it superlative ratings among museum exhibits, including “Y es, it's 
informative. The magnitude, the size of the globe is impressive. This is one of the best 
exhibits right now,” and “Best presentation I have seen in any museum in quite some 
time.” Others spoke of how the information came to life. “Definitely - makes history and 
general conditions of earth come alive,” and “Absolutely. Brings science to life.” 

The most common responses were that the Sphere is interesting and informative. A few 
of these many comments include, “Absolutely - it's phenomenal,” “Yes, superb 
instruction,” “Yes - extremely enlightening and educational. This variety of information 
is amazing!,” “it is much easier to understand complex ideas with visual pictures and a 
live explanation,” and “Yes, because it is very informative. This sphere that constantly 
changes, goes back in time, can predict the weather, changes from earth to Mars and I've 
never seen anything like it before. It’s awesome.” 

Others elaborated on the educational value of the Sphere in being able “to see these 
things up close. A typhoon actually forming, landmasses forming, differences among 
planets, etc. Great idea for presenting a lot of science,” and to help “people to actually 
fee! the changes that planet has gone through.” 

Several said they wanted to return to learn more. For instance, “Too much to see in one 
visit,” “a lot of information to absorb - would retain and learn more if repeated,” “[would 
come back] just to keep learning, lots of information, and it was very cool to watch,” and 
“Would come back as often as possible. Wonderful learning experience.” 

Visitors felt the Sphere provided a valuable learning experience for children and adults. 
They said it was “Very educational for all ages,” “[an] excellent tool for children. Brings 
all the data to life,” and “This is excellent material to understand the earth and Mars! 1 
am not good at the science, but it's easy to understand for me! You should show all 
children in the world. Everyone loves it.” Others mentioned with whom they would 
return. “This time I would bring children especially, so that they could see how 
everything is connected,” and “1 would bring my Earth Science class to see this.” 

There were a few negatives. One visitor said, “It is entertaining but not factual.” And 
another said they would come back, “but would prefer home computer version.” And one 
student said he wouldn’t return “because we do not need to know all that stuff yet.” 
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Learning 

When asked about what they had learned, several individuals in both written surveys and 
interviews following presentations were unable to specify particular things they learned 
or overwhelmed. Some of these comments include, So much - absorbed like a sponge - 
will have to take time to decipher,” “Quite a bit too much to list. Maybe a prop(s) to help 
illustrate the scale - the earth is so large and the time covered so vast. It is a GREAT 
IDEA/exhibit,” and “Too much to list here. The earth to sun perspective. Information on 
Mars. Formation of the various countries. Explanation of why the crust moves.” 

Some provided lists of what they had learned and mentioned disparate ideas from several 
different data sets. One wrote “Clear view of Pangaea and movement of plates through 
time. Weather near the South Pole is really turbulent. South Korea has a lot more 
electricity than North Korea. Denver was underwater around the time of the dinosaurs,” 
and another,” and another recalled the following: ”1 never realized the degree of 
movement of the continents. That there's water on Mars [at] the Polar Ice Cap. Some of 
the details - long canyon, high mountains, hot spots.” One woman contrasted the things 
her daughter learned, “ about Mars and weather on earth” and the things she had 
learned,” about the sun's weird rotation pattern.” 

The animation of plate tectonics was again mentioned by a large number of visitors, 
followed by global warming, weather, and features of the solar system. Visitors 
comments suggest that they learned big ideas, mechanical explanations, and gained an 
ability to visualize global views, time and scale, and scientific events. These are 
described below, with examples drawn particularly from the plate tectonics animation. 

Big Ideas: Visitors gained an appreciation for the earth as dynamic, and made comments 
such as that “The Earth is always changing,” and “The earth is constantly evolving.” 

Mechanical Explanations: Visitors commented on gaining new understandings of things 
like how the continents formed, “Seeing the continental drift and the "million years ago 
countdown" - it explained the concept to me, for once it was clear!,” that “ The world is 
liquid rock and how the world was long ago,” “The way the continents moved through 
the years and what happened to the life on Earth,” and “Why Mt. Everest is getting 
higher.” 

Visualization of global views: Visitors noted visualizing a variety of global views and 
processes. This included global views of geographic features, “Pacific Ocean covers 
nearly half of the planet,” storm patterns, such as “liked seeing the storm patterns - can 
see where the storm activity is. Parts of the world get more, I can’t say enough about it,” 
and “Didn’t know where all the storms came from, where weather patterns come from,” 
among other images. 

Visualization of time and scale: Several people noted that they were unaware of plate 
tectonics prior to Pangaea, “Didn’t know there was a continent before Pangaea,” and 
others noted the extent of continental movement, “1 never realized the degree of 
movement of the continents,” and “Earth movements over time have been much more 
dramatic than I realized.” Visitors noted learning about scale in relation to some of the 
planetary comparisons, “I learned how big earth is compared to the sun,” and they also 
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realized the scale of global warming, “I learned about the degree to which the Earth is 
getting warmer, which was the most impressive for me.” 

Visualization of significant events: Visitors noted that the Sphere helped them to 
visualize a variety of events, including ‘‘what earth was like long ago and prognosis to 
come and why,” “What land masses looked like in prehistoric times,” “Extinction of 
dinosaurs. It was very photogenic in regards to how comet hit the earth. Easier to 
understand when you saw it on the globe,” and “How the world was 600 million years 
ago and also where Indiana was located.” 

Particular facts and specific examples: There were numerous facts and specific 
examples mentioned by visitors, including “That India "crashed" into Asia,” “That the 
Earth got as cold as it did during the Perma-extinction,” that “Indiana used to be 
underwater and the Niobra Sea was much bigger than I thought,” and “that Mars has a 
North Pole made of ice.” 

In some of these cases, visitors were struck by aspects of the visuals that were not 
specifically articulated by presenters, such as the amount of plate movement over the 600 
billion years of Earth’s history, while in other cases they picked up on particular details 
mentioned by the facilitator. For instance, individuals mentioned the formation of the 
Himalayas, watching Indiana move, India crashing into Asia, following Hurricane Isabel 
and the size of the Valles Marinas canyon. 

Interestingly visitors consistently incorrectly remembered the time span of the plate 
tectonics animation. They consistently wrote that the animation depicted 600 million 
years, when in fact it depicted 600 billion years of continental drift. It is unclear what 
exactly led to this confusion. 

Comments and questions during and after the presentations suggested how closely 
visitors were listening to the presenter, and following the imagery on the Sphere. For 
instance, a little girl asked at the end of one presentation whether Mt. Everest will ever 
get high enough to fill the Marianas Trench, piecing together disparate bits of information 
provided in the presentation. 

Clarity 

In response to a question whether there was anything confusing in the presentation, the 
responses were overwhelmingly no. Many stated that the presentation was dear. In every 
presentation, visitors freely asked questions, and some even commented on written 
surveys that they had no questions because they had been able to ask them directly. “The 
presentation was not confusing — answers (to our questions) were immediate” A teacher 
noted, “The presenter answered questions well and gave good information. It was clear. 
He respected kids’ questions and had a good understanding of kids.” 

In a few cases, visitor comments suggest that they were initially confused by a visual, but 
a satisfactory explanation had been provided by the facilitator. These comments include, 
“Weather patterns - but it was explained,” “Temperature rise because of C02 over the 
next 1 40 years - but then he explained it more.” 

Two visitors noted that the representation of elevation on the Mars image was counter- 
intuitive and though they understood the image, felt it was worth pointing out. “One 
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image of Mars looked reversed. Meaning high areas looked depressed and low stood 
out.” 1'his was the only misleading visual noted by visitors. 

There were a few remaining questions which visitors indicated on surveys. One visitor 
wrote “El Nino- What is it exactly?” and “What was confusing was the thing of the 
hurricanes.” Another visitor seemed to want more in-depth explanations, and wrote, 

“Yes. About as informative and simultaneously confusing as weather descriptions on TV. 
Too much, too fast, without making clear what is happening, what the causes/effects are.” 

A few visitors questioned some of the scientific data presented, particularly the 
reconstruction of continental drift reaching back 600 billion years. For instance, 
“"Scientific' 1 theory of continental drift reaching back more than 200 million years seems 
to be reaching too far.” 

One noted that the presentation would have to be changed for children, “The information 
shared verbally was very geared to adult language and understanding. To make the 
exhibit more interesting to kids the information needs to be refined some to their level.” 

Suggestions for Modifications 

In an effort to see how visitors could imagine changes in the presentation, or if they 
considered other interactive modes for the Sphere, visitors were asked what they would 
do if they could control the images on the Sphere. “Would they slow them down, speed 
them up? And what else would they like to see?” 

There was little consistency in their responses. Visitors were divided between those who 
said the image speed was good, or wanted either faster or slower images. In some cases 
they suggested that some images should be slower and others faster. There were a 
handful that requested that the images of weather be speeded up to highlight patterns, 
such as: “I would speed it up, especially the weather to see patterns or irregularities. It 
would be interesting to have more pictures to see different occurrences at different times. 
One visitor suggested slower animation in the recent years of Earth’s evolution. “I would 
like to see the images change over shorter time periods as it nears present day (i.e. - 
instead of 1 million year increments, go by 50K increments for past few million years, 
then 5 K increments, so can see effects of recent ice ages, human development, etc.” 

Visitors suggested a number of other modifications to the images. These included 
repeating images, tilting images, and zooming in or augmenting with illustrative images. 
These are described below. 

Repeating Images: A few mentioned the value of repeating or stopping animations, as 
presenters did, frequently in the tectonic plates demonstration, but in others as well. For 
instance, “Replay them and include maybe some more pauses.” 

Tilt: Others noted the ability to tilt images and suggested incorporating this effect to a 
greater extent. “Be able to tilt the images on the earth's axis for a different perspective.” 
“Rotate the image so all sides are seen.”” Bring the poles to the side (so equator runs N to 
S) for some of the views.” This was in fact done in the presentation on Earth’s 
topography. 
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Zoom: Several recommended zooming in on geological features or providing additional 
information through dose up shots. A few of these comments include: “I would zoom in 
to see evolution and weather,” "Could one zoom in on various parts of the planets? 
Heavenly body anatomy from core outwards - possibly on display on walls as posters or 
diagrams,” "Zooming features - zoom in while rotating. Zoom out while rotating,” and 
“Any possibility to zoom in? That would be interesting. I like geography so continent 
identification is possible.” 

Height of Sphere: A few viewers suggested placing the sphere closer to the floor. One 
explained that this would make the Northern Hemisphere easier to see. 

Selecting Images: Only one visitor suggested including the ability of visitors to select 
images. 

Content Suggestions: Visitors made a number of suggestions of additional images they 
would like to see. These included requests for additional planets," How storms and other 
weather patterns develop, more detail of ocean depths,” "Ice cap melting and coastal 
changes,” and “real-time earthquakes.” Also mentioned were “global warming trends 
going back in history,” “images of life on particular continents,” “View of spacecraft in 
space near earth," “more about U.S. geography," and “projected continental drift.” 
Another visitor suggested that “It might be interesting to see how pollution travels around 
the globe and ozone depletion.” 

One of the visiting teachers suggested she “would like to see pictures about under sea 
volcanoes and landforms, fish related to particular undersea landforms. Connect the 
Maryland curriculum and images they are used to with what they see on the globe. What 
would landforms actually look like?” 

And several visitors indicated that there was “Too much to list.” 

Other imaginative suggestions from viewers included “If the sphere could grow into 3- 
D . . . .” and “Imagine an all-solar system built with spheres.” 

Satellites and Imaging 

There was a striking absence of mention in the surveys of either the satellites which had 
been used to collect data or the production of the data sets in use. This was despite the 
fact that the satellites were mentioned in almost every demonstration, and in some cases 
different satellites and kinds of data were discussed. Further, a handful of comments 
suggested visitor assumptions that real time data is accessible for weather and other 
displays, Visitors appear to take the availability of this data for granted. The realism of 
the images may contribute to this view. The only data set which visitors even mentioned 
was the entirely computer-generated animation of continental drift - one which visually 
looked like an animation rather than a realistic image. 

These observations suggest some challenges foT presenting the Sphere and possible 
content for accompanying kiosks or materials. Both the challenges to collecting 
appropriate data and the manipulation of the data to create realistic looking images could 
be addressed. During a 25-minute presentation for a school group, many of the students’ 
attention began noticeably to drift during an explanation of satellite and data collection. 
The images on the Sphere are so compelling, it will be a challenge to deliver information, 
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such as that about the satellites, that can hold up to the images on the Sphere. Visual 
images of the satellites might be one way of reinforcing their importance and developing 
a more lasting impression. In fact, a MSC staff member suggested hanging “a 
geosynchronous satellite at appropriate distance from Sphere.” 

Presenters noted NASA’s Blue Marble inaccurately presents an image of Earth entirely in 
daylight, in contrast to the representation of the Earth in which one side is in daylight and 
the other in darkness. Information about the different kinds of satellites, photography, and 
imaging used in creating the images could all be addressed in ancillary panels or kiosks. 
For instance, supporting information might include examples of various forms of satellite 
photography, computer graphics (plate tectonics), color highlighting of topography, and 
discussion of the various ways in which models represent and distort the truth. 

Conclusion 

Comments throughout the surveys reflected visitors’ enjoyment of the experience and 
suggest that it was a rich and successful learning experience. Visitors described the 
exhibit as “cool,” “riveting,” “fantastic,” “fascinating,” “magical,” “phenomenal,” 
“informational,” “educational,” and as the “best in the museum.” Visitors appreciated the 
Sphere as an aesthetic experience, a piece of innovative technology, and as a compelling 
and versatile educational tool for children and adults. Visitor engagement in the 
presentations was high. 

The Sphere powerfully demonstrated the different sides of celestial bodies, such as near 
and far sides of the moon; and Earth’s hemispheres in darkness and light. In all 
presentations, viewers were impressed by explanations of relative size, such as 
comparisons of the Valles Marineris on Mars, and the Grand Canyon on Earth, and of the 
sizes of the Earth and Sun. 

The presentations were effective in conveying a range of levels of scientific ideas and 
facts, from big ideas about the Earth as dynamic, to memorable details such as the 
poisonous gases of Jupiter. Visitor responses to the presentation of scientific information 
on the Sphere suggests its strength for introducing global contexts for understanding local 
issues, understanding global interconnectedness of a range of social and natural 
phenomena, and presenting global transformations through time. 

Visitors commented on the variety of data presented and the many possibilities for other 
presentations. Teachers imagined a number of possible uses of the Sphere, including 
conveying ideas in Earth Science, Chemistry, Biology, Astronomy, and Geography. And 
had novel suggestions from integrating animated characters to designing activities using 
global positioning systems. 

Visitor expectations of what kinds of imaging can be shown are high. In several cases 
visitors seemed to assume that real time data is readily available. They could also easily 
imagine the representation of a wide range of geological and social images and processes. 

Visitors greatly enjoyed the live presentation, praised the knowledgeable presenters, and 
the opportunity to ask questions. The presentations were well-crafted; information needed 
to explain the images on the Sphere was an integral part of the narration. Furthermore, 
elements like color were effective in highlighting topography, changes in sea 
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temperature, and other features. The only visualization which visitors continued to 
struggle with was the representation of altitude on the image of Mars. However, without a 
live presenter, there may be much more need for supporting information. 

In all but a few cases, visitors did not seem concerned by their lack of opportunity to 
control the Sphere themselves. On a few occasions, when the group was small, visitors 
were privileged to a more interactive experience, and presenters pulled up data sets in 
response to particular questions and interests. 

Science centers may want to consider automated demos. Depending on the complexity of 
the text, these would likely require arrows or boxes on the projected images to draw 
viewers’ attention to local events and features. This would of course, take away from the 
aesthetic experience and sense of transport. 

There were as well visitors who enjoyed the aesthetics of the Sphere and found it restful. 
If the Sphere is located where visitors are passing by it might be valuable to have data 
sets that are understandable and enjoyable without a presenter, and with no or minimal 
supporting information on kiosks or displays. This might also be a time to give visitors 
the opportunity to control the images on the Sphere in a limited way - selecting from a 
choice of easily understandable data sets, such as Earth’s topography, and allowing 
visitors control of the speed of the images. 

From comparisons of the 3-D view to those of ordinary 2-D views, to descriptions of 
seeing the view of an astronaut, visitor comments suggest that the Science on a Sphere’s 
imaging provided a provocative and challenging experience. Viewers experienced a fresh 
perspective. A number of visitors further saw scientific and political significance in this 
global perspective. 
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Appendices 

Appendix A: Questions 

Interview Questions 
Survey Questions 

Appendix B. Data 

I. Interview Data 

• Visitor Survey A 

• Visitor Survey B 

II. Surveys 

• Survey A 

• Survey B 


RMC Research Corporation: Science on a Sphere Front-End Evaluation DRAFT 


20 



90 


Appendix A: Questions 

Interview Questions 
Survey Questions 
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Science on a Sphere: Visitor Survey (Interviews) 

Data Collectors : Complete this section during the presentation. Date 

Check ALL of the topics covered in the presentation 

Human impact on earth 

Earthquakes, volcanic activity, plate tectonics 

Earth’s changing atmosphere, climate 

Data Collectors: Ask visitors the following questions. Try to capture their words exactly. 


Solar system 
Oceans and weather 


Introduction: Science on a Sphere is in the design phase. We’d like to get your feedback to 
help develop the permanent exhibit 

How old are you? Male/Female 

How would you rate this exhibit? l=poor 2=fair 3=average 4=very good 5=excellent 
What did you like most about this exhibit? (ask for details) 


Did you like the presenter and style of presentation? Why or why not? 


What did you think of the content of the presentation? Was it too simple or too complex? Was the 
information new to you or did you know it all already? Was there too much or too little 
information? 


Is there other kinds of information you needed to make sense/use of the experience? 


Did you like the length of the presentation? Was it too long or too short? Did the images move 
too quickly or too slowly? Would you have liked more time just to look and watch the images? 


Describe the image you saw that is most memorable to you. 


Would you come back to see the Sphere again? Why or why not? 


RMC Research Corporation: Science on a Sphere Front-End Evaluation DRAFT 


22 



92 


Science on a Sphere: Visitor Survey (Interviews) 

Data Collectors: Complete this section during the presentation. Date _________ 

Check ALL of the topics covered in the presentation. 

Human impact on earth Solar system 

Earthquakes, volcanic activity, plate tectonics Oceans and weather 

Earth’s changing atmosphere, climate 

Data Collectors: Ask visitors the following questions. Try to capture their words exactly. 


Introduction: Science on a Sphere is in the design phase . We’d like to get your feedback to help 
develop the permanent exhibit. 

How old are you? Male/Female 

How would you rate this exhibit? l=poor 2=fair 3=average 4=very good 5-excellent 
What most impressed you about the Sphere and presentation? (ask for details) 


Is this like anything you’ve seen before? How is it like other things, how is it different? How does it 
compare to seeing the same science presented on a map or a film or video? 


Did you feel confused by what you were seeing at any point during the presentation? If yes, please 
explain. What kind of support could be provided so that visitors are not confused? 


Tell me what you think you learned in this presentation (about climate, plate tectonics, etc) 


Do you think this is a good environment to view the Sphere? Is it quiet enough to focus? Too busy? Too 
many people in the group? 


Would you recommend the Sphere to others? Why? 
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Survey A 
Visitor Survey 
Science on a Sphere 

Check ALL of the topics covered in the presentation 

Human use of electricity Solar system 

Earthquakes, volcanic activity, plate tectonics Earth’s weather 

Earth’s geography Global warming 

Check your Age: Under 18 18-25 26-49 50 and over 

Circle: Male or Female 


How would you rate this exhibit? (circle one) poor fair average very good excellent 


Describe the most memorable image you saw. 


If you could control the images on the Sphere, what would you do? Would you slow 
them down or speed them up? What else would you like to see? 


What do you think is the point of the Sphere? How is seeing things on the Sphere 
different from a map or on video? 


Would you come back to see the Sphere again? Why or why not? 
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Survey B 
Visitor Survey 
Science on a Sphere 

Check ALL of the topics covered in the presentation 

Human use of electricity Solar system 

Earthquakes, volcanic activity, plate tectonics Earth’s weather 

Earth’s geography Global warming 

Check your Age: Under 18 18-25 26-49 50 and over 

Circle: Ato/e or Female 


How would you rate this exhibit? (circle one) poor fair average very good excellent 


What most impressed you about the Sphere? 


Was there anything you saw that was confusing? Explain. 


What did you learn? 


Would you recommend the Sphere to others? Why? 
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Appendix B. Data 

I. Interview Data 

• Visitor Survey A 

• Visitor Survey B 

II. Surveys 

• Survey A 

• Survey B 
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I, Interview Data 


Interviews were conducted with groups of school students who attended Science on a Sphere 
demonstrations. Two fourth grade classes and one sixth grade class attended presentations. At: 
each of these a handful of adult or family visitors also joined the presentations. 
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Visitor Survey A 

Demographics 

Visitor Survey A: Eight groups were interviewed using this survey, with a total of 
approximately 30 individuals. This included four groups of 4 th graders; two groups of 8 th 
graders; one group of young adults (one male and one female), and one family group with four 
children and two adults. 

Rating 

Three groups rated the sphere very good, three groups rated the sphere excellent and two groups 
of children didn’t answer. 

What most impressed you about the Sphere and presentation? 

4th graders 

• night and day, Volcanoes, Earth movement. Dead volcanoes on Mars 

• weather, looks real, old to new/current earth, learning the size of the earth and sun 

• earth in real size, shows the planets in relation to earth, shows how earth looked 600 
million years ago 

• when the earth was breaking up, 600 million years — 5 day, hurricane Isabel , pretty 

sun 

8th graders 

• weather, climates, different features. Mars and Jupiter, global warming, plate tectonics 

- gave them a better visual idea of lessons learned in class 

• the images (Earths 600 million year storm - 1 like the sun) Warmer/Pangaea/The sun 
Family Group/ Adults 

• El Ninos heat and what happened, earth changing 4 million years ago to now, 
visualizing, weather - 4th grade 

• you feel like you 're looking at the earths movement, colors , can see it happening, real 
time data but like data, would be bored looking only at Mars 

Is this like anything you've seen before? How is it like other things, how is it different? 
How does it compare to seeing the same science presented on a map or a film or video? 

4th graders 

• TV weather, cooler rotation, see whole planet, more than weather, lots of stuff 

• no. colors. When we look with our eyes we see a different image. It looks so small. Map 

- to show all colors and hear they change and move 

• globe moves around, better on globe/can see it ’s not flat 
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• In books - in Social Studies book, the planet 
8th graders 

• no/changes in front of you, vegetation, 3D - lots of different information at the same 
time- 8th grade 

• on the news - on the computers - a lot better - not too long - much better 
Family Group/Adults 

• no, yes at zoo, globe of earth 3-D but doesn 't change, moves, can walk around , span of 
time, can see parts come together 

• like an IMax - gives you a different view or scale - nothing like it before 

Did you feel confused by what you were seeing at any point during the presentation? If yes, 
please explain. What kind of support could be provided so that visitors are not confused? 

4th graders 

• not confusing, answers were immediate 

• Jupiter - had different patterns 

• confused by different colors 

• some of the words, continental drift, he used examples 
8 th graders 

• No, he explained everything seen before in Science, might be confused if he wasn ‘t here 
presenting 

• No, no problem 
Family Group/Adults 

• explained well 

• was pretty clear 

Tell me what you think you learned in this presentation (about climate, plate tectonics, etc) 

4th graders 

• earth rotates, day and night changes, planet Mars has volcanoes and canyons, planets 
have volcanoes (dead) 

• being an astronaut allows you to see these things, how it rotates, electricity at night - we 
use a lot of lights, volcanoes in the ocean 

• how planets changed, land is not flat but 3-Dimensional, learned about the other planets 
in the solar system 

• 600 million year ago - that the world would look like that , making the dinosaurs, air on 
Jupiter was poisonous . The sun is bigger than the earth. 
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8th graders 

• Jupiter - gases movement patterns. Mars - volcanoes, Sun -flares, didn ’t know how it 
affected the earth vegetation, effects of sun and meteor that killed the dinosaurs 
(Pangaea) 

• learned how the earth ’s changed, explosions on the sun, earth temperatures 
Family Group/Adults 

• El Nino - how to read colors of presentation, storms come up from S to N, start in 
Africa. Didn 7 know there was continent before Pangaea. 

• 1st -person - Didn 't know where all the storms came from, where weather patterns come 
from. Also, changes of earth over billions of years, images of Mars and Sun. 2nd person 
- Temperature of water and global patterns, gulf stream. (This was in the context of a 
discussion of the popular film, The Day After Tomorrow.) 


Do you think this is a good environment to view the Sphere? Is it quiet enough to focus? 
Too busy? Too many people in the group? 

4th graders 

• Liked moving around the globe, fun to look at 

• yes. Lighting quiet, not busy, size was good 

• ok 

• [they liked] when it was moving and changing 
8th graders 

• too much light, announcements kept interrupting, should keep in the museum, another 
school group will change their experience, recommend keeping small groups together 
from start to finish 

• needed to move around, about the right size 
Family Group/Adults 

• Saturday may be hard, lights - hard to see on side, will be distracting 

• lot of activity to the employee lounge? But kids exhibit didn 7 phase them, background 
light is distracting, darker atmosphere - looks like it is floating. Love the blue marble 

Would you recommend the Sphere to others? Why? 

4th graders 

• y es 

• yes. learn about the earth, educational, weather, learn how weather moves, for example 
hurricanes came from Africa and go to the East coast. 

• Two said yes and one said, l want to come for my birthday 
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• I'd tell my mom, tell my friends 
8th graders 

• Yes .1 liked how it looked 

• yes. It was awesome, coolest, the best 
Family Group/Adults 

• My kids held on for 30 minutes. The adults were OK with the length. (Would recommend 
images of the) Sun, Mars, Earth - millions of years ago 

• Yes. It was very visual. I have visual friends, deaf friends. [The Sphere] is unique for 
getting a world wide perspective, though it’s always good to point out particular places. 
[In the animation of continental drift] show Maryland not Indiana 
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Visitor Survey B 


Seven groups total were interviewed with these questions. These included three groups of 8 th 
grade students and three groups of 4 th grade students, with three to six children in each group. 

In addition, one group of adults (2 women in their twenties) answered these questions and a 
group of 4 th grade teachers were interviewed. 

How would you rate this exhibit? l=poor 2=fair 3=average 4=good 5=excellent 

Five groups rated the exhibit as excellent (though one member said poor), and one group 
thought the exhibit was very good. One group was divided, and 4 of the members said the 
exhibit was excellent, while three said it was very good. One group thought the exhibit was very 
good. 

What did you like most about this exhibit? (ask for details) 

4th graders 

• get more information than a book, glows and spins, the information changes [ variety ] 

• NA 

• seeing the weather - seeing the sun and planets - seeing earth like an astronaut - seeing 
how much electricity U.S. uses - land changes over time 

• They were into it. Even some of the more active ones. 

8th graders 

• showed earth clearly, little things make sense, showed how planets look 

• everything was detailed - colors; that it was accurate, especially plate tectonics. Get to 
see how the Earth looks- colors, storms, movement, and predicting [weather] 

• revolving - different points of view - latitude and longitude -where storms came from 
and went, global look 

Adults 


• Educational, see things 3-D. Liked storm cycles; Paleo/ evolution of earth; never knew it 
was one clump, never knew they broke apart. Good speed, but good to have it repeated. 
Good to get out of the North America - the storm systems begin in Southern hemisphere. 
It shows that what happens on other continents affects us. 

Did you like the presenter and style of presentation? Why or why not? 

4th graders 

• explained very well, helped understand the content 

• cool presentation, magic turning the earth 
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• Teacher: The presenter answered questions well and gave good information. It was 
clear . He respected kids ' questions and had a good understanding of kids. 

8 th graders 

• he was good, it was good, could ask questions 

• good, clear 

• cool - like presenter - explains it well - like ability to ask questions 

Adults 

• If more kids - more involvement. Some big words for kids. We explained, ok for her 

What did you think of the content of the presentation? Was it too simple or too complex? 

Was the information new to you or did you know it all already? Was there too much or too 

little information? 

4th graders 

• Wanted to zoom in close to see me, see the moon, current weather. Already knew the 
information about planets/space. Information about weather, hurricane Isabelle was 
new. 

• earth spin no confusion; cool about volcanoes - didn ’t know they were dead 

• Teacher: content appropriate and sufficient for age group 

• NA 

8th graders 

• just right content [for the students]; some of the information was new, information was 
alright 

• just right. Some information was new, such as that storms come from Africa. Would like 
to see more planets - Saturn, Pluto would be interesting. 

• Fit what they talked about, in school. Some was new, such as information on global 
warming. Knew about Earth ’s formation billions of years ago, but it was new to see it. 
Also new was how the earth looks at night, and how people use electricity. 

Adults 

• good info, just right, length ok 

Is there other kinds of information you needed to make sense/use of the experience? 

4th graders 

• symbols - what different colors meant, direction. His explanations put information in 
context, for instance, explanation of Pangaea. 

• too much information — want to come back for the presentation 
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• Teacher suggests more lessons/more information - lecture on all four oceans, harming 
continents. Make it more interactive, visual aids of dinosaurs etc for changing earth. 

8th graders 

• no, maybe show volcanoes when talking about volcanoes 

• Organized, but would like to know more about the moon. 

Adults 

• could take some images and make panels to show what the graphics are 

Did you like the length of the presentation? Was it too long or too short? Did the images 
move too quickly or too slowly? Would you have liked more time just to look and watch 
the images? 

4th graders 

• wanted to hear more, length was just right 

• it wasn 't long - but lots of stuff 

• Teacher: the length was Ok, would have liked more time to watch weather and how 
specific continents changed 

8th graders 

• four said it was good, two said it was too long 

• just right length, speed good 

• length was just right - would like seats, really cool, yes would have liked more time just 
to watch the images, but the pace was good 

Adults 

• no fine, repetition good 

Describe the image you saw that is most memorable to you. 

4th graders 

• Two liked Jupiter and noted the colors up close, and two liked the Sun. One said it was 
because they didn 't know about Sun 's gases and another that they like sunshine 

• the sun has volcanoes and explosion - Jupiter (poisonous gas) 

• mars - weather, astronaut view of planet - 
8th graders 

• Mars, Jupiter 

• How plates move, how countries changed and split apart, how earth looked in morning 
and at night, how it looked millions of years ago, how they showed global warming 
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• storms and hurricanes, plate tectonics starting 600 billion years ago, global warming, 
lights at night 

Adults 

• way earth formed —formation of continents 

Would you come back to see the Sphere again? Why or why not? 

4th graders 

• 3 yes - want to hear more, go on longer, it was fun , learn more; 1 no - it made you 
sleepy 

• Absolutely ! # or longer time next time 

• yup 
8th graders 

• yes -if I need to, no - kids won ’t be interested, only if they changed the content 

• I liked how it looked, liked information, interesting 

• yes — interesting, well explained, would recommend 
Adults 


• would take people to see it, cool to see weather up to the minute 

Additional Teacher Comments 

• For elementary students it s s hard to visualize 3D when you are used to 2D. The Sphere 
made it seem real because it can tilt the earth and rotate. One of the children said its like 
I 'm an astronaut. 

• The teacher said she would like to see pictures about under sea volcanoes and 
landforms, fish related to particular undersea landforms. Connect the Maryland 
curriculum and images they are used to with what they see on the globe. What would 
landforms actually look like? 

• It 's amazing. It was helpful. You feel like you were an astronaut. You got the experience 
of a Jim Lovell or an astronaut looking out at the Earth. 

• Particularly liked seeing the storm patterns - can see where the storm activity is. Parts 
of the world get more, I can ’t say enough about it. 

• Good. Higher level for their kids, but good to get different type of exposure. The model 
of Earth and tracking storms is good for them.. They 7/ remember the eye of the storm. It 
was a great experience. 

• The ■ visual is good for them. To see from the astronaut s view. What do they [astronauts] 
see? That s a question I’ve had in class... Seeing night time and day time is interesting. 

• Seeing the structure of the US is good. This is neat. 

• They were into it - even some of the more active ones. 
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II. Surveys 

Two versions of the print survey were distributed during Phase Two of data collection for the 
evaluation (June 1 8-30). Both surveys shared a similar format, with one closed question asking 
visitors to rate the exhibit followed by four open-ended questions. 

Rating (combined totals from both Surveys A and B) 

Across all age groups, visitors rated Science on a Sphere with extremely high ratings. Out of 
326 visitors, 67% (219) rated the exhibit as Excellent. Thirty-one percent (100) rated the Sphere 
as Very Good. Six Ninety-eight percent of visitors rated the sphere as either Very Good or 
Excellent. Two percent (6 visitors) rated the Sphere as Average and less than 1 % rated the 
Sphere as fair. 



Fair 

Average 

Very Good 

Excellent 

Total 

Under 18 

1 

3 

28 

42 

74 

18-25 

- 

- 

12 

16 

28 

26-49 

- 

3 

36 

88 

127 

50+ 

- 

- 

24 

73 

97 

TOTAL 

1 (>1%) 

6 (2%) 

100(31%) 

219(67%) 

326 
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Survey A 

Describe the most memorable image you saw. 

Venus (5) 

Mars (5) 

The Sun. (2) 

Indiana forming over 500 million years. 

All of the images were fascinating. 

Weather. 

All of them were great. I really enjoyed the weather image. 

The way the continents moved through the years and what happened to the life on Earth. 

Global warming. 

When he showed global warming over a few 100 years. 

Lights at night and global warming. 

The continental drifts. Just seeing them. This is 3-D. (vs. map) Books are books, you see it 
(passage of time). This is modem, advanced. 

In darkness, the lights of the most populated areas on Earth. Also, how the current day 
continents formed. 

The shifting of the land/water from millions of years ago to now and the 1 cm per year 
separation. 

To actually see weather systems in real time. 

Earth at night. 

Earth’s continental change. 

Global warming - continental drift. 

Continental drift. 

El Nino 

Plates moving - land forming. 

Formation of the Himalayan Mountain. 

Everything. 

Weather on Earth 

Hurricane Isabelle Plate Tectonics (boundaries, ocean trenches, and spreading systems) El Nino 
and La Nina were excellently displayed. Continental drift with vegetation changes!! 

The earth at night and when the Himalaya's formed 
Global warming. 

Lots of scientific information. 
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The rotating Earth. 

Plate tectonics. Mars. Hurricanes 
Entire presentation was interesting. 

Sphere spinning - 3-D image and also large. 

The Earth. 

Movement of structures over time. 

Everything. 

Creation of land masses over time - weather wealth of country - due to electricity, light, weather 
over time. 

Today's weather. 

Weather changes, Earth's geography. 

Would like to see real time all around 3-D ball. 

Satellite imagery with Hurricanes. 

Electricity. 

Night time and movement of continents. 

Rapid onset ice age. 

Use of electricity. Planet lights up. 

Earth. 

The global warming timeline and Mars. 

Global warming. 

Plate tectonic movement. 

The loop of the continental drift over 500 million years. 

Migration of tectonic plates. 

Complete surface of planets in motion/Earth, Mars, Jupiter. 

The warming of the Earth. 

Geographic changes over millions of years. 

Earth's geography 
El Nino. 

Cross between Earth and Mars. For earth, you get so caught up in your life - 1 live in MD. When 
you see the Earth so far away it puts life into perspective. You see the whole thing you think 
about all the wars going on and you say.... Book is flat. This is 3-D and puts it more visually 
into perspective. When we look at Mars it's interesting because of the activity going on with the 
Rovers. Keeps up on the news, internet and paper. 

Continental drift. 

Plate shifting. 
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Global warning. 

The continents separating. The different ages of the Earth. 

Found them all interesting. 

Earth as from space and cloud coverage. 

Pangea's expansion. 

The view of Earth from space. 

Global wanning. 

Weather images. 

Blue Marble. 

The change of millions changes 
Io and sun (x-ray) 

Pangaea to present day. 

Continental drift. 

India crashing into Asia. 

The weather image to predict hurricanes. 

Weather model in near real-time. 

Hurricane Isabel 

The lights around the world. 

Ocean void of human touch - man made objects. 

The way it made facts and theories clear. 

Storms forming in Africa. 

Drift. 

The heating of our planet. The movement of our continent. 

Plates moving over 600 million years. 

No - most - each was incredible for its uniqueness - making other contrasts more evident. 
Global warming. 

Temperature changes 

Isabel - movement of continents. 

The Earth forming - different countries. 

Formation of continents. Especially India's movement to Asia. 

Needs to be bigger - needs formal shows - chairs, more theatre-like setting. 

Continental shift. 

1 think it becomes very tangible to see the actual planets rotating. 
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All. 

Changing geography in time. 

Moon craters from volcanic activity. 

Weather. 

Human use of electricity. 

Formation of current Earth. 

Pangaea 

The million year formation. 

The movement of the continents over time. 

Global warming. 

Continental drift. Weather/El Nino, Mars and Sun. 

Plate tectonics, when we saw Indianapolis move over the time of millions of years. 
Very good learning experience for my grandchildren. 

600 million years to present! Time-warp - one of the best learning tools I've seen. 
Seeing the continents change. 

Earth 600,000,000 years ago. 

Time change of continents 
Formation and track of Hurricane Isabel. 

Continental drift; global wanning. 

Formation of the continents. 

Continental drift. 

Formation of the continents. 

Formation of India 

Continental drift. Night view showing electrical use. 

Global warming. 

Earth - Pacific Ocean. 

Global warming. 

Changes of the land mass over the years. 

Global warming - night skies. 

Night lights. 

The making of present day. 

NASA view of daylight - Earth. 

Jupiter, green sun. 
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Global warming. 

The changing of the earth. 

El Nino 

Weather around the continents. 

The green picture of the sun I saw. 

All. 

Seeing the formation of Earth crumpling and stretching until it reached present day. 

The most memorable image was rather a series of images - the animation of plate tectonics and 
continental drift through millions of years. It gave an easily understandable, broad view of 
continental drift. 

When I saw the continents forming. 

The current weather and how the hurricanes come about. 

The light brightnesses. 

Indiana moving. 

India crashing into Asia. The change in the continents. 

The storms coming across the ocean. I also liked the lights across the globe. 

That of global warming on the Earth. 

Real time weather. 

Hurricane Isabel. 

The satellite picture, the Blue Marble. 

The image of global warming. 

El Nino 

Global warming in 2250. 

Mars Grand Canyon 5 times larger than the Earth's Grand Canyon. 

Hurricane Isabel. 

Mars, Hurricanes. 

Nothing stuck out. 

Hurricanes. 

The earth changing over the years and the lights at night. 

The Earth moving. 

The moon IO. 

The image of the Earth continent formation Pangaea - current especially Indiana's location. 

The image on global warming. 
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If you could control the images on the Sphere, what would you do? Would you slow them 
down or speed them up? What else would you like to see? 

It was good./alright/keep it the way it is (22 mentions) 

Slow down. (16) 

No (7) 

Appropriate speed was demonstrated./Good speed. (6) 

I would speed it up. (4) 

It was perfect speed. To see what more the future will bring. 

Replay them and include maybe some more pauses. 

The images were shown at a good enough rate, (would like to see more about U.S. geography). 
How things were formed. 

Nothing - it's fine the way it is. 

Excellent rotation speed. The remaining planets would be nice. 

Slow down - show meteor impact 65 million years ago. 

Keep it moving slow. 

I would keep it the way it is. 

The sphere should be lower so the northern hemisphere can be seen better. 

Rotate the image so all sides are seen 

Stow them down at some points but the person giving demo did this. She did a great job. Could 
one zoom in on various parts of the planets? Heavenly body anatomy from core outwards - 
possibly on display on walls as posters or diagrams. 

Good speed - small group was able to repeat several functions. 

I would slow them down. I would like to see the climate control. 

Bring the poles to the side (so equator runs N to S) for some of the views 
Slow down. Place the sphere lower to the floor. 

Images of life on particular continents. 

It was fantastic. 

Real time weather. 

Both slow and speed in different spots. Slow at the dinosaur killer. 

Zooming features - zoom in while rotating. Zoom out while rotating. 

Slow down - just right. 

1 would not change a thing - 1 found it fascinating. 

Imagine an all-solar system built with spheres. 

Speed up slightly, more on severe weather. 
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Neither, protected areas. 

Slow them down especially the timeline. 

More stable projector towers and sphere. Perhaps an automated demo. Plus more seating around 
the sphere (off the pods). 

It might be interesting to see how pollution travels around the globe and ozone depletion. 

If the sphere could grow into three D. 

Would like to see weather patterns sped up. Too much to list. Best presentation/demonstration 
seen in a long time. 

Could have different scenarios to look at. It could prompt you to go to other. . . 

Speed was good, would like to see Jupiter. 

Speed, as most of the audience will be nominal scientists. 

Weather effects. 

Time lapse. 

Real time earthquake, weather patterns, etc. 

Same darker room. 

Global warming trends going back in history - warming in this century. 

OK for what was watched. 

Speed was perfect. Be able to tilt the images on the earth's axis for a different perspective. 

Slow them down; ocean floor, SST hotspots, continents over time. 

Speed is just right. How the temperature has changed from the past to the present. 

I would like to see the images change over shorter time periods as it nears present day (i.e. - 
instead of 1 million year increments, go by 50K increments for past few million years, then 5 K 
increments, so can see effects of recent ice ages, human development, etc. 

It was great. Would be neat to be able to select images yourself. 

Slow them up. Different planets explained. 

I'd slow and speed them up. You can learn from both. 

Speed perfect - Images perfect. 

Slow for some - fast for others. 

Venus - Barth's weather. 

It was excellent. 

Slow down - the color coding helps perspective. Any possibility to zoom in? That would be 
interesting. I like geography so continent identification is possible. 

Ice cap melting and coastal changes. 

Forward past current time. 
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Good speed. Other evolutionary changes - such as shift from aerobic to anaerobic. How storms 
and other weather patterns develop, more detail of ocean depths. 

The Earth transformation and cooling. 

Change to the planets. 

View of spacecraft in space near earth. 

Other planets. 

Ozone depletion. Fast forward in time. 

Very well paced. Well covered. 

Speed was OK - the outer planets. 

Slow down and decrease remote delay to be able to see more detail. 

Slower - seats around 

Don’t know enough to reply seemed fine to me as is. 

Slow down Pangaea - continents movement. 

Our moon. Projected continental drift. 

Slow them down and get a view of my state. 

I would show the major cities. 1 also would slow it and make it fast. I would also like to see a 
birds eye view of the world. 

Nothing, same, and the same. 

No comment. 

I would slow them down. The labels of cities, other planets. 

The images were fine as is. 

I would zoom in to see evolution and weather. I would keep it the same speed. 

I would speed it up and I would show every planet. 

I would like to see how it was different along time ago to today. 

Slow them down. The outline of plates. 

I would like to see more information about weather patterns, more NASA images. 

Pluto. 

Same. 

Well presented and informative. Loved it. 

I would slow them down to see the effects on the Earth. 

I would slow the sphere down a tad. I would also like to see other planets. 

I would leave it the same, however I would lower the sphere. 

Speed it up and see meteors in space. 

Slow them down. A close up image of all the planets. 
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The Pangaea splitting. (He told me he didn't understand where the clouds went. He understands 
the concept.) 

Perfect rate of speed. The moon. 

Speed them up (as an addition to watching slow). 

1 would keep it the same, maybe expand a little more on each topic. 

I would speed it up, especially the weather to see patterns or irregularities. It would be 
interesting to have more pictures to see different occurrences at different times. 

Animal evolution. 

What do you think is the point of the Sphere? How is seeing things on the Sphere different 
from a map or on video? 

3-D (6) 

3-D help to bring out visuals better. 

To teach us learn so much more! 

It is 3-D and more interesting than a book or map. Easy to see weather patterns, etc. 

To show the Earth as it is. 

Global perspective is new and important in a nationalistic society. 

Realism. 

It gives you a real look at the earth. 

To show how the Earth has changed. It makes it much more interesting and it shows it in a way 
like no other. Very interesting. 

It makes the images normally portrayed in a flat, 2-D manner more tangible, thus making them 
more real. 

The point or the Sphere is to give the audience a 3D image of the information being relayed. It's 
more lifelike. 

Brings it more alive. 

Gives you the whole picture. 

More realistic. 

To give you a 3-D image of the Earth, etc. Easier to visualize. 

To represent various interactions (weather, electricity, topography) involved with the planet. 
Show the relationships of events on Earth. 

Sphere more realistic - better perception of Earth. 

It makes the images more relevant. 

Things are in 3D, movement is realistic, gives true perspective. 
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See the whole world. 

Gives a world view of geological phenomena. 

This information. 

Interactivity. 

See world and universe from a different perspective - more 3-D! 

Show movement of weather continents 

The 3-D view. It gave me a new impression of the Earth. 

Understand land formation, weather. 

One is really seeing the true shapes and sizes of images opposed to flattened distorted images 
found on 2D maps and computer screens. 

Very realistic - Can't possibly present this on a flat surface - i.e. night and day. 

Education, on the Sphere - it's much more realistic and a great visual representation 
La Nina 

Easier to comprehend in 3-D model. 

It brings the image to life. 

It's cooler. More people can watch 
It is much rounder. 

Different perspective. 

It gives a different perspective that you can't get from other media. 

It reflects how alive our planet is. 

Present Earth/planets/Sun as really exists vs. 2-D image. 

More realistic. More 3-D. More details. 

To show as close to actual what is happening. 

Sphere was very helpful to understand. It felt like a real one/Earth. 

You get a three dimensional view. It is visibly very stimulating. 

Excellent visualization of changes in earth over time and many other ever-changing variables, 
such as weather patterns. 

To give a more accurate perspective of the topics covered. 

Yes. It's easy to understand. 

Gives 3-D image, makes it more graphic. 

How dinosaurs became extinct. 
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The sphere gives you a more interactive experience (visual). 

3-D and motion. Ability to show various directions of motion - creates real effects. 

Much more comprehensive and tangible. 

Sphere is a more realistic presentation of information therefore enhancing learning. 

To show different areas of the sphere as it turns as opposed to seeing it on a map or video when 
it is still and not a three D form. 

Total Earth perspective. 

This is much better than a map - also stimulates discussions. 

3-dimensional is more interesting. 

To explain atmosphere changes in the Earth and land changes. As well as the environment on 
different planets. 

To understand how connected the earth is; to demonstrate that to people. The sphere allows you 
to see the interaction of weather around the globe. 

3D always gives a true life perspective to the demo. 

More visual. 

I think it is a great educational tool. 

Gives full view of planet weather, plate shift. 

More of a realistic educational tool. 

Easier to understand. 

The same. 

It is very instructive to see 3-D images on a sphere, which cannot be accurately understood or 
appreciated on a 2-D surface. 

Perspective and holistic experience and interpretation. 

The sphere is more realistic - overall good presentation. 

3 -dimensional provides realism. 

Makes it more realistic. 

Realistic. 

Sphere approach makes presentation much more actual - as the components of the solar system 
really are huge steps beyond flat screen approach which we’ve seen all our lives. 

Educational. So much more graphic. 

How much Earth was covered with water when the meteorite fell. 

Shaped like Earth - realistic perspective of relationships. Shows human impact. 
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The forming of the Earth but 1 know that Mother Earth came into creation. Almighty God 
Jehovah is the creator of all things. 

Speaker to have a microphone to better hear him. 

Conservation! 

Very much easier to understand. 

More realistic. 

Much more realistic. 

You get a more global view (no pun intended). 

Changes of Earth. Sphere more graphic/amazing. 

It gives a more realistic impression of the planets, moon, sun, etc. 

More real. 

Gives you more of a 3-D effect. 

Allow you to visualize with more perception. 

More vivid. 

The 3-dimensional effect makes it more real for you to understand the concepts. 

A lot clearer. 

Education. 

More true images, much better perspective. 

Much better orientation with sphere. 

You can view the earth in its totality. 

It adds appropriate perspective. 

Yes, the narration. 

In motion and shaped correctly made it easier to understand 
I liked it very much. 

It shows dynamism, complexity, weather and economic development. Very informative and 
insightful. 

Very educational and historical. The sphere gives a great 3-D effect. 

More realistic. 

Yes. 

A more realistic 3-D effect. 

More accurate view. 
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To see the Earth as a globe. 

3-D effect, use of motion. 

Better and more realistic understanding of the formation of the planets and the systems - 
weather, geography, future, etc. 

Realism. 

To see the world in 3-D - more easily understood. 

The sphere makes learning about the Earth's history much more interesting than a map or video. 
It's a great educational tool. 

Great teaching tool! 

Like sphere - didn't like having to move - wanted to sit down. 

To show about the Earth look better and gives more information. 

You can actually look up at it or see the details. It's better than 2-D. 

To show what Earth looks like, maps don’t' show as much. 

The sphere is like the diagram of the Earth with more detail than certain maps or videos. 

Very good for visualization. 

It's more like real life with the sphere. 

Gas, clouds. Because you can get a view of the whole planet. 

You feel like you're in space looking down on the Earth - more physical feeling. 

Better visuals. 

It's a lot easier and fun. 

So the person can show all around the world. I like the sphere more because it moves. Second 
you can answer questions and ask questions. 

On a map or video, you can not walk around the image and the image isn't distorted. The point 
of the sphere is to teach people about earth in a new and innovative way. 

The sphere gives a 3-D visual of everything on the globe, not just a portion. 

To understand how things have changed. 

You don't have to use you're hands to turn it around. 

Y ou get to see things like they are. 

The point is to witness how different things are impacting the Earth in a positive and negative 
way. 

Way cooler - more interesting. 

The point here is to help you understand what we have to look for in the big picture. 

RMC Research Corporation: Science on a Sphere Front-End Evaluation DRAFT 49 



119 


You get some exercise! 

It allows you to view the Earth as it is in 3-D. Not just on a map. 

Seeing things on the sphere is different because you can get a better idea. 

So you can understand. 

The sphere shows you a lot more. 

You can actually see all around. 

Because it is turning. 

The sphere makes it easier because it is how the image is meant to be viewed, not exaggerated 
on a map. 

More realistic Earth picture. 

It is a wonderful way to see the planets 3-D. 

Because it shows actual size. 

You get to see around object and it seems more real. 

To teach people about Earth - it's kind of like the Earth so it teaches a bigger lesson. 

To show changes on the Earth. The Sphere shows more details. The Sphere makes it easier to 
see where things happen. 

It shows how things work on the world from a new perspective. The point of the sphere is to 
create a better image. 

It looks more real. 

Map doesn't show other ways around. Globe shows it better than a flat map. 

All the ways to map things on Earth. 

Would you come back to see the Sphere again? Why or why not? 

Yes. (38) 

Yes, because it is very interesting. (13) 

Yes, very informative. (7) 

Y es, just to keep learning, lots of information, and it was very cool to watch. 

Yes, it was great! 

Y es, excellent presentation. 

Yes - excellent presenter - very knowledgeable - Expert. 

Yes. It was very educational on a variety of related topics. Will bring family. 

Yes, it was very modem and technology in science is always beneficial. 

Y es, I learned a lot about the way the earth was fonned. 
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Yes! I would recommend to others. I think you could do a simpler presentation for children. 

Yes - 1 definitely learned a lot of information. 

Yes. It was the most interesting exhibit. 

Yes because it's interesting and covers several subjects in a short period of time. 

Yes - a lot of information to absorb - would retain and learn more if repeated. Enjoyable to 
watch. 

Yes. It is full of information you do not see anywhere else. 

Yes! I would bring my Earth Science class to see this. 

Absolutely - it's phenomenal. 

Absolutely - amazing. 

Yes - interesting facts well presented. 

Fd come back if more images were added - other planets? 

Yes. I would come back to see the sphere. I think it gives a fascinating vision of our other 
worlds. 

Definitely, I will bring my family, out of town visitors. 

Yes, to learn more! 

Yes. Presentation was excellent. 

Yes, I loved it. 

Yes, superb instruction. 

Yes, very informative and interesting. 

Yes. Like to see more. 

Yes. Could be expanded into more in-depth presentation, very informative. 

Yes, missed first part. Always catch more if viewed more than once. 

Absolutely. Brings science to life. 

Definitely, yes. The 3-D images on a sphere are unique and explain and show effects that cannot 
be seen in 2-D. 

Yes. I was extremely enlightened. 

Yes, the amazing growth over time. The growth of the planet. 

Yes - would like to see more data sets. 

Sure, can always learn more about our world. 

Yes, but not from here. 

Yes. Much more can be done with this. Real time earthquakes, etc. 

Yes. 1 think you should have a cocktail partly around it. It’s very engaging. It moves. 

Yes. Too much to see in one visit. 
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Yes. It taught me a lot about how the shape of the world is changing. 

Yes. To see additional projections. 

Yes. It is much more realistic than a map. 

Yes. To see other things while the presenter is pointing out the main features and to see if 
anything is new. 

Yes, it's interesting and unique. 

Yes, it was real great. 

Loved the intimacy of the seating with sphere. Moveable platform floor. 

Yes. Definitely. This time I would bring children especially, so that they could see how 
everything is connected. 

Yes, very educational. 

Yes, yes to learn more. 1 might be better to curtain it off from rest of exhibits. 

Yes, if there is an excellent demonstrator such as we heard. 

Maybe if I was here again. 

If other planets were added as a different presentation which also included Earth and Sun maybe 
same are with addition of Earth's moon. 

Maybe. 

If we were local, we'd be back - from Connecticut. 

Yes. Much to learn. 

Would come back as often as possible. Wonderful learning experience. 

The education - movement/commentary. 

Yes - bring my children and grandkids. 

Yes. To show someone else. 

It is entertaining but not factual. 

Yes - not enough time. 

Yes, definitely. 

Yes, but would prefer home computer version. 

Yes, I would. Information will be evolving as we leam more. 

Yes. To absorb more. 

Yes. Yes. Yes. 

Yes - lots to leam. Our tax dollars at work well. 

Yes, very informative, educational. 

Yes. Fun and easy to leam from 
If it's in my hometown California. 

RMC Research Corporation: Science on a Sphere Front-End Evaluation DRAFT 52 



122 


Y es. Riveting. 

Definitely - makes history and general conditions of earth come alive. 

Yes! Great. 

Absolutely! 

It was very educational. 

Yes, I'd be interested to catch the entire presentation. This time I'd be able to formulate 
questions, as I'd have some knowledge of the sequence of the presentations. 

Absolutely - it's a great teaching tool. Very realistic. 

Yes - fascinating to see these things up close. A typhoon actually forming, land masses forming, 
differences among planets, etc. Great idea for presenting a lot of science. 

Yes - the Earth is constantly changing. 

Sure. 

Yes - to see changes in the Earth's atmosphere, land, etc. due to global warming. 

Yes! I only stayed a few minutes and loved it! 

Yes, learning and humorous host. 

Yes I would come back again if I could. I would come back because it's very interesting and you 
learn a lot. 

Yes, to learn more. 

Yes. It is very neat and it teaches a lot. 

Yes, it was fun and cool. 

I would because it was funny and educational. 

Yes, it's just cool! 

No, because 1 already know what happened. 

Not just for the sphere but I would come back. 

Yes, because it was totally great. 

Yes. I think it's fun to watch. 

Yes, 1 enjoyed seeing the entire globe at once. 

Yes, very interesting plus being able to see the Earth and the Sun. Very neat idea. 

Possibly it depends on who I'm talking to. 

Yes. It was very educational. 

Yes, it’s bitchin’. 

Yes, it's really cool to see the Earth moving and to see the predictions of the weather. 

I might because the presentation had lots of interesting facts and pictures. 

Yes I would, because even though I knew a lot, I learned a lot. 
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Yes! Because il is so cool. Very good presenter! 

Yes, because il is cool. 

Yes. Way cool! 

Y es I would, it was very educational, and the guide was great. 

I think so, I think it's very interesting and I like the way it's set up. 

Y es, because I really enjoyed the demonstration. 

Of course, to see any additions or just to view it again, it's truly a spectacular idea. 

No, because we do not need to know all that stuff yet. 

Additional Comments 

This is a compilation of the feedback from a group of 4 people who fdled out surveys, but were 
not as detailed in the written responses than they were while talking to me. This group spent 
about 25 minutes speaking to Dr. Bendel and though it was the best exhibit in the building. They 
really enjoyed the personal instruction/explanation they received and would not have preferred a 
formal presentation. They also were very impressed with the new technology involved in this 
exhibit as well as the presentation of the sphere and how it seemed to float in mid-air. One man 
suggested that the sphere would be really effective in a theatre, such as the planetarium. They 
also enjoyed the interactive nature of the exhibit, (e.g. the presenter being able to call up images 
they wanted to see.) Personally, I think (and 1 have gotten similar feedback from others) is that 
the exhibit itself is great, the sphere is a unique way of presenting the information. What we can 
improve is the presentation of the information - more speakers, or at least a better way for the 
speaker to address more of the audience at once. 

1 didn't realize there was the large mountain on Mars or the big crater. You could have the 
Magic School Bus which just had an episode on Venus, for school presentations. - and to reach 
the age of kids like her 5 year old. 

This is a wonderful exhibit and would be an asset to the MD educational system which focuses 
on Earth Science as well as biology and chemistry. If possible this would be an excellent 
addition to specialized (science/academic) H.S. that already contain in-house planetariums. I 
actually like this better because one could present these programs in addition to constellations. 
Best in the museum. 

Would be good if it was lower. 

There were some terms that weren't explained. 

Yes. To bring others to see it - it's fascination. 

Great! Thanks - very up to date. 

Phenomenal. 

I did not view the entire presentation. 
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Survey B 

What most impressed you about the Sphere? 

Everything. (4) 

Realism. (4) 

Resolution explanation. The resolution is fantastic. This kind of globe in Terralink, but this is 
much better. Explanations were appropriate and informative. 

Plate tectonics, hurricane movement. 

The technology of the presentation. The rotation is very cool. 

The video’s. 

The 3D demonstration, decent explanation, plate tectonics of global warming. 

Video quality. 

The ability to see everything to scale, the oceans, land, etc. 

Amount of knowledge about all topics. 

Watching the land mass change over 600 million years and the animation technology. 

How it changed 600 million years ago. 

The time lapse earth. 

The continual global motion keeps the audience interested. 

The way it showed the earth at night and how the light on earth can be seen in space. 

The continental drift part and how the sphere accurately showed all of it. 

Cloud cover, electricity. 

The effectiveness of its ability to express data. 3-D presentation put the information in context. 
The whole history of evolution. 

Liked the day and night lights, moving land, pictures in general. 

The way it could be used to show various data sets in their native global format. 

The continual motion. 

The realism of the various data sets as they were represented on the sphere. 

Simply the idea of it - never seen anything like it! I‘m not even much of a science person - it’s 
just very cool. 

The dynamics of setting it up and getting it perfectly placed to effectively show the presentation. 
How it works! Very cool. 

History of plates. 

Yes. 

Linearity of display. 
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Continents moving. 

The presentation was very thorough. The human use of electricity was mind blowing. 

Resolution was good. Very effective demonstration. Better seating and less light would prove 
helpful. 

It's a great visual learning tool. 

Color/active. 

Seeing electricity use, plate tectonics, evolution of earth's surface, other planets and sun 
Constant change of the different screens projected. 

Great lecturer. (Will was giving demo - Geoff). 

The time lapse visualization in 3 dimensions. 

Indiana moving. Mars, sun information. 

Clarity of detail of images. 

Visualizing images 3D. 

It was a neat way to present the data. Kids liked it too! 

Clarity, the 3D aspect which is very informative. 

We could see the earth millions of years ago and see how the earth changed. 

The pictures. 

Amazing technology, our beautiful home Earth. 

The visual made the explanation very easy to grasp. 

Seeing whole planet at once. Night scene - population location. 

Graphics. 

The electricity usage imbalance. 

Variety of data sets. 

Unique presentation, extensive databases, knowledgeable staff. 

The real life feeling of seeing the areas of the earth all at once. 

Different detail - like seeing it from space. 

Real time weather phenomena around the globe. 

Ability to visualize effects on a global basis. - The big picture. 

Attention grabbing - stands out. Clarity of image - brightness of projections. 

The presentation. 

Gives you a very different perspective on weather and plate tectonics that is very entertaining 
and informative. 

Everything. I'm impressed about the thing talk I wouldn’t have even thought about. It's wild— 
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Much more dynamic than a flat presentation, e.g. noting side of storms. Who doesn't like 
looking at their home. 

Representing planet earth in its actual. 

Plates in motion! Excellent! I'm a visual learner. I didn't realize there was that degree of 
movement. 

Extinction of dinosaurs. It was very photogenic in regards to how comet hit the earth. Easier to 
understand when you saw it on the globe. 

The amount of scientific information that was presented. 

Size and the movement. 

Ability to switch views and interactively observe events and objects, planets, and their 
atmosphere. 

With the Earth suspended in air it felt as if I was outer space looking in. 

3-D view and interchangeable screen 

The variety of information and data that was shown. The 3-dimensional spherical view was very 
enlightening, seeing weather, topographical, and continental shifting was most interesting and 
the presenter’s knowledge and experience was extremely interesting. 

The ability to show many different topics and time periods. 

Computer technology brings 2 dimensional to 3 dimensional for children. 

The multiple functionality. To see Isabel weather data. To see the sun and Mars. 

Computer coordination to show the image. 

Unique methods of visual display. 

Beauty, reality, content, storytelling by the presenter. Comparisons. History. Scale. Simply 
awed. 

Its movement and color and size. 

Good discussion, interesting graphics, it's a great teaching tool. 

I liked how it showed the movement of the earth’s crust. 

The images of the changing topics. 

The historical background. 

The picture of the earth. 

The amount of water on earth, identifying countries by electricity. 

The view, the suspended aspect. 

Shows how earth looks from space and great facts about drift. Mars, Saturn and the sun. 

Jt s so amazingly beautiful and we are SO small in the scheme of things! 

The changes in time. 

Visualization and time evolution of important events. 
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The change the sphere makes and how it can go back in time!! 

It gave us a real feel for the shape and events happening on the surface of the planets. 

Can't pick just one thing. 

Movement. 

Very good presentation. 

Clear, concise. 

Very knowledgeable. 

The whole program. 

Enjoyed the human element of a moderator. Clear presentation. 

The beautiful colors like an abstract painting. The globe is gorgeous to look at. 

Ability to show how the continents developed. Severity of global warming. 

The technical aspects. Magnificent - how do they do it? Spent some time trying to figure out 
how it’s done. 

How the projections were done on a round object. 

The commentator was excellent and knowledgeable 
Excellent, easily understood visuals. 

Easily understood concepts. 

The gentlemen's presentation. 

The perspectives over time and the spatial perspective (the big picture). 

Versatility of what can be shown on the sphere. 

Graphic, clarity, 

Movement of the continents to create what exists today. 

The graphics and ability to cover so many topics and make it visually clear. 

That 2 thin wires could hold a sphere that size. 

Animation, time tracks, color. 

Versatility. 

Detail. 

Clarity and realism. 

The presenter's knowledge and ability to explain to the audience. 

Very impressive - operation of sphere - projectile. 

The pictures and the explanations. 

How it looks like it rotates. 

Clarity, visibility. 

Learning about how the world was like long ago. 
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How it showed the world and how it was 600 million years ago. 

The accuracy and current weather forming. 

How you project it. 

It seems to be accurate. 

Small reptiles turn in to big reptiles. 

The accuracy. 

A lot of stuff is shaped like a sphere. 

Visual rotation/tilt/animation with a good explanation. 

The 600 million years ago part. 

How the pictures get on the sphere. 

The part when he said the nickel is as big as the earth. 

How it was exactly the same tilt as the earth. 

Global warming. 

It's size. 

The graphics, and the continental drift. 

Traveling into the future to see what would happen. 

The way it shows how time evolved. 

The second to last slide. 

Going to other planets. 

How you can project something onto a sphere. 

How you are able to see the planet's rotation and geography. 

How the earth moved over 600 million years. 

That you could get a spherical view of all the planets which could mean that it is the most 
accurate map you'll ever have. 

The facts were very accurate. 

The technology is amazing! 

The details in everything. 

That it can cover everything you talked about. Different because it can spin around and show 
you the whole thing. 

Was there anything you saw that was confusing? Explain. 

No. (77) 

Nothing. (3) 

Temperature rise because of C02 over the next 140 years - but then he explained it more. 
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What was confusing was the thing of the hurricanes. 

No, explained very well. 

The deviation/depression scheme on Mars. (He asked about it) 

No, very explanatory. 

Forming of earth. 

No, everything was coherent. 

Not really, for an informal introduction to this information/data. 

It's a lot of information but I felt I could absorb it at my leisure. 

Outstanding presenter. 

Interesting - easily explained for the non-scientist. 

None. 

No. Not really. 

Plate tectonics would have been great in reverse! 

South Pole has severe weather. 

If we know about global warming why isn’t anything done about it. 

Weather patterns - but it was explained. 

The information shared verbally was very geared to adult language and understanding. To make 
the exhibit more interesting to kids the information needs to be refined some to their level. 

No. Explanations were clear. 

No, it was all explained very clearly. 

Some shimmering effects on the bottom of the globe especially when data was in motion. 

No. The presentation was informative and engaging. 

No, well some. 

He explained everything well. 

Scientific theory of continental drift reaching back more than 200 million years seems to be 
reaching too far. 

No. More information? 

No, everything that I saw was explained very thoroughly. 

No - the presenter did an excellent job! 

One image of Mars looked reversed. Meaning high areas looked depressed and low stood out. 
Good explanation. 

[The presenter] was very clear in his presentation. 

El Nino- What is it exactly? 
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Yes. About as informative and simultaneously confusing as weather descriptions on TV. Too 
much, too fast, without making clear what is happening, what the causes/effects are. 

N/A 

The presentation helps focus your eye and mind. Even so, it is awesome even if you are just 
looking at it. I liked the complexity, whoaa NOAA! 

No- all questions were answered fully. 

No. Exhibition was quite explicit. 

Weather (hurricanes). 

None - the lecturer made everything very clear. 

Although I didn't see the whole presentation, it was very clear. 

I didn't know why the weather turn green. 

None. 

Nothing particularly. 


What did you learn? 

A lot! (8) 

How the continents formed and a lot more. 

Geography of solar system. 

Even when you sleep the world stiil goes on. 

Pacific Ocean covers nearly half of the planet. 

I learned about the degree to which the Earth is getting warmer, which was the most impressive 
for me. 

Fonnation of hurricanes, crusts, continents. 

Global warming. Different rates of rotation [of the gases] in the sun. El Nino- what and why and 
many others. (This was fantastic. Never thought the results could be felt in such a short time 
frame.) 

Too much to write here. 

A lot - mostly as supplemental to a college and high school basic education. 

N/A 

What the changes look like continuously. 

That the sphere is the first spherical movie screen. 

About the solar system's highest mountain. 

How the earth's land masses (continents) were formed. 

Mars crater (Magellan?) is as wide as North America - was caused by volcano action raisins 
crust. 
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Clear view of Pangaea and movement of plates through time. Weather near the South Pole is 
really turbulent. South Korea has a lot more electricity than North Korea. Denver was 
underwater around the time of the dinosaurs. 

What NOAA does, what it researches, scientific theories of past and predictions for future of 
planet. 

The presenter was an excellent teacher. 

Continental plates and how they moved and a couple causes of extinction. 

Explained dinosaur era, what earth was like long ago and prognosis to come and why. 

The unfortunate impact of global warming and the plate tectonics data set was impressive. 
Effects of El Nino, Mars topography, weather on earth. 

That India crashed into Asia. 

Maybe a prop(s) to help illustrate the scale - the earth is so large and the time covered so vast. It 
is a GREAT IDEA/exhibit. 

Mostly the shifting and creating of the different continents. 

That NOAA is interested in educational outreach. 

Reality. 

Seeing the continental drilt and the million years ago countdown - it explained the concept to 
me, for once it was clear! 

That the Earth got as cold as it did during the Perma-extinction. 

Mars geography, sun rotates. 

Global warming, surface of Mars, weather. 

Too much to list here. The earth to sun perspective. Information on Mars. Formation of the 
various countries. Explanation of why the crust moves. 

A lot of things I forgot since high school. 

The movements of the lands. 

What land masses looked like in prehistoric times. 

I learned how the continent was changed. I learned the global wanning. I learned Mars has a big 
canyon and some water. I learned that California may be separate from North America but 
connect again. 

Plate tectonics in motion. 

1 learned more about how the continents actually moved about to fonn the landmasses we know 
today. 

How the correct continents formed face of earth/sun. 

I could easily follow the presenter over a widely different subject. 

I learned about the planet Mars and its atmosphere. 

Formation of continents and continual change currently. 
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Cool graphics. 

My daughter learned about Mars and weather on earth. I learned about sun's weird rotation 
pattern. 

Underwater crust, Andes Mountain, hurricanes 
The more I know the less 1 know. 

About sun. Mars, the height of the volcanoes - they length of the 

Use of energy resources, movement of continents, weather movement. 

About moisture and weather radar. About solar weather. 

Lots - weather patterns, continental shifting, population/light density, and more. 

Surprisingly a lot. Mostly about earth transformations, Mars geography, Saturn. 

History of our planet and it's possible fiiture. 

Martian geography was great. 

I never realized the degree of movement of the continents. That there's water on Mars - the Polar 
Ice Cap. Some of the details - long canyon, high mountains, hot spots. 

Lots about the planets - very cool. 

How weather is decided, different depths of world. 

This is a great way to explain and show things to kids. 

How clouds moved around Jupiter. 

Indiana used to be underwater and the Niobra Sea was much bigger than I thought. 

The weather how it can actually start in Africa and what is seen from space. 

How the present continents formed over millions of years. The shape of the continents and 
weather information in relation to hurricanes, typhoons, etc. 

Quite a bit too much to list. 

Global weather patterns. Hurricanes develop. 

A lot more about different aspects of lite on the earth and some possible consequences. 

History of earth. 

A lot about our planet now and the future. 

Past and present helping speculating future of planet earth. 

The way weather patterns move from the equator to the hemispheres. 

How earth moved. The 4 projectors give very clear view of how continents have formed. Better 
than reading, even 3D Imax. I really liked the formation of N. India, Himalayas. 

Refresher course. 

I was impressed by all of it. 

1 did not realize that the movement of the earth was so great. 
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How far the sun is from Earth, How the earth looked millions of years ago. 

The lengths of the periods in time. 

The earth is dynamic. 

Information about Mars. 

Too much to write. 

Too much to say. 

The earth is constantly evolving. 

Earth movements over time have been much more dramatic than I realized. 

El Nino, Continental Drift. 

About dinosaurs and their extinction, mantle of earth, shifting of continents, climate changes 
over time. 

Oh my - mainly visualization of creation of earth's current geography. 

Movement of continents, seriousness of global warming. 

Formation of the continent. 

So much - absorbed like a sponge - will have to take time to decipher. 

How global wanning is a reality and will have scary consequences. 

Everything was shown. 

Why dinosaurs disappeared. 

How the mountains were formed, why the continents move. 

Visual explanations ofhow earth evolved. 

Plate tectonics. That it got to -100 degrees in the Permian. 

600 million years ago there was not anything green. 

Plenty. 

History and development of planet. 

Too numerous to list and brought sense to all science classes learned in school and college. 

I didn't realize how the earth is in motion. 

A whole bunch. 

Continental shift, ice caps on Mars, weather patterns. 

How the earth is constantly changing. Mt. Everett still growing. 

A great deal. 

How the continents formed - the movement was far superior to still pictures. The oceans are 
growing wider. 

Too many things to list - this was great! 
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Dynamic earth. History. Plate tectonics. Vast changes. The eyes of science can tell great stories. 
Scale e.g. sun vs. earth. Changes over time scales e.g. Global Carbon Change, Pock-marking by 
, meteors. 

About global warming and earth's geography. 

That the weather changes in big ways. 

How hot it would get in 500 years. 

I learned how global warming will affect the earth in 500 years. 

Too much to say. I liked seeing how the colors showed what he was talking about. The valley 
and 3 volcanoes and big volcanoes showed elevation. 

1 learned that Mars has a North Pole made of ice. 

India and Madagascar had an affair. (Not real answer!) Mars' geography. 

1 learned where Indiana was years ago. 

I learned about the drifting continents. 

Why Mt. Everest is getting higher. 

Planet, and sun information, weather patterns. 

Why tectonic movement is continuous. 

How the earth is warming more and more. 

1 learned how big earth is compared to the sun. 

1 learned that the world won't stop changing in a couple of years, but not for a long time. 

The Earth is always changing. 

That the world is made of liquid rock. 

Mars has a bigger mountain but we (earth) have more ice. 

The world is liquid rock and how the world was long ago. 

Lots - earth's geologic history was especially good. 

Nothing. 

How storms form, when and where. 

I learned all about earth. 

How the world was 600 million years ago and also where Indiana was located. 

Everything presented. 

How the earth evolved and the weather patterns, 

South America, North America, and Africa were one big island long ago. Also, Mars has an ice 
cap at the very top of it and it is made of ice. I learned many other things like that Mars has the 
tallest mountain and it has the deepest and longest canyon. 

In 500 years we'll have global warming. 

About the earth. 
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Electricity can make the earth hot. 

How electricity makes the planet hot. 

How plate tectonics work. 

Would you recommend the Sphere to others? Why? 

Yes - extremely interesting./ very interesting. (8) 

Yes, very informative and interesting, (6) 

Yes, it was very informative. (4) 

Yes, because it's awesome. (2) 

Very much. It was very informative on different topics. Fd mention giant, 6' spherical 
projection, changeable globe that goes through different topics. 

Yes, I thought this was one of the coolest things I have seen. Excellent, 

Yes. (19) 

Yes - unique way to see the earth. 

Definitely - it was interesting, educational and in an intriguing form. It kept my attention - I'm a 
humanity's major. 

Yes, easy conceptualization or abstract concepts. 

Yes, you learn a lot about the past that make things the way they are now. 

Y es. Visual learning. 

Yes, because it was very interesting and you can learn many things. 

Yes, it can be a very interactive experience to teach people about the earth they inhabit but don't 
know much about. 

Yes. I think having a moving sphere and a person to explain is better than the static models. 

Yes, I would because this is a great experiment to know. 

Yes. It explains a lot and actually shows what happens. 

Yes. Interesting to see. 

Definitely, especially to high school students and middle school students. 

Yes, very educational. 

Yes - kids stayed interested - could understand easy. 

Most definitely. It’s an interesting, informal way to introduce various information about ours and 
other planets, moons, the sun. Would like to see more data sets. 

Yes, because it gets you up and about to view the various quadrants of the sphere rather than 
just lectures you while you sit in one spot. 

Yes. Please make this a permanent exhibit. 

Yes. It makes information easy to see. 

Absolutely, very educational. 
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Yes - very good information. 

Yes, marvelous and intriguing display. Every planetarium or high school needs one orthese. 
Yes, great geographical description of continental drift. 

Y es. It's also very relaxing to just look at it. 

Greatly! It was enjoyable. 

Y eah. It's great for teaching. 

Y es, it's marvelous to see it all laid out before you. 

Yes, so they can understand the weather channel a lot better 1 do. 

Absolutely. 

Absolutely. I think kids could be inspired and their curiosity piqued with these demonstrations. 
Very impressive - unique view of the earth. 

Absolutely - what a great format. It's an amazing way to image things. Young children may 
need some way more to interact with the lecture. 

Yes! This is excellent material to understand the earth and Mars! I am not good at the science 
but it's easy to understand for me! You should show all children in the world. Everyone loves it. 
Sure. It was cool to see what astronauts see. 

Yes! What a terrific learning tool! 

Definitely, yes. 

Yes, awesome, simple enough to understand. 

Yes, very interesting exhibit and presentation. Presenter was very knowledgeable about 
subjects. 

Yes - more interesting to see spherical features on a globe instead of a flat panel. 

Yes, a new way to learn. 

Yes - an excellent teaching tool. 

Yes - very detailed - see how our earth looks from a different perspective. 

Yes, it's informative about the planets Mars and earth, etc. and weather. 

Yes. Totally different than seeing in 2-D. Asks; So Europe is really a different continent than 
Asia. Provides grounds for theory of Asia as separate continent. 

Yes - very different - 1 liked the sphere. 

Absolutely. Best presentation 1 have seen in any museum in quite some time. 

Absolutely recommend - 1 hope you make it permanent. 

Very exciting exposition and educational too. Highly recommend it. 

Absolutely yes. Because it's interactive. I thoughl il was good. 

Very educational for all ages. 
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Absolutely! Very visually stimulating from a macro/God’s-eye view of the world. 

Yes, very educational. Helps people to actually feel the changes that planet has gone through. 
Yes - extremely enlightening and educational. This variety of information is amazing! 

Most definitely - you learn, and retain, so much more by seeing what you are learning. 

Yes. Neat to see the whole world in many ways. To see whole world weather. Very detailed. 

Yes, excellent tool for children. Brings all the data to life. 

Yes, it's informative. The magnitude, the size of the globe is impressive. This is one of the best 
exhibits right now. 

Yes. It is easier to view by a larger group of people. You can cover a wide variety of topics. I 
liked being able to walk around the sphere to get different perspectives. I like the global view as 
if from space. 

Yes! Informative and interesting. Very good lecturer. 

Yes - exact representation of earth. 

Science class high school. 

Yes - bring it back. 

Yes, very informative and interesting. 

Yes - very interesting. Educational. 

Let it stay. 

Awe inspiring. Nothing else like it. Reality is better than anything that fiction can invent. It's not 
in schools. And needs to be out in front of the public and school children. 

Very impressive display. 

Yes, it is much easier to understand complex ideas with visual pictures and a Jive explanation. 
Yes - to give a good perspective of where we live. 

Yes, very informative/enjoyable. 

I think it's good to know. 

Yes! Great Show and very informative. 

The interpreter was excellent!! He is very knowledgeable and communicates very well. May be 
a waste of time with no interpreter. 

Yes, 1 would. I feci this demonstration can help foster a greater desire in our youth, especially to 
study the changes in our world. 

Yes. Generally interesting. 

Yes, because it is very informative. This sphere that constantly changes, goes back in time, can 
predict the weather, changes from earth to Mars and I’ve never seen anything like it before. It's 
awesome. 

Yes. Great background information. 

Yes, very visually informative. 
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Yes! Explains so much. 

Yes, because it is loaded with wonderful information. 

Absolutely - for all ages! 

Yes - the educational value. 

Yes - fascinating - relevant. 

All education can set you free. 

Yes, very educational. 

Most certainly - shame this is here after school's are closed - every age group - children through 
adults can learn. 

This is the best exhibit in the museum. 

Yes. It gives a new perspective on earth. I will tell my friends not to miss it. 

Yes, because of its excellent quality. 

Yes, very educational. Excellent presentation! 

Yes - clear and comprehensible and broadly informative. 

Y es - Interesting and educational. 

Yes, fantastic source of information. 

Yes, because it will help us to learn that we have to stop causing danger to the earth. 

Because it's cool and factual. 

Sure.... why not. 

Yes, because it's interesting and it teaches new things. 

Yes, amusing and educational. Narrator.... entertaining. 

Yeah, it gives you a better understanding of our everyday world that most people don't 
understand. 

Certainly, very educational. 

Y es, because it was a really neat exhibit. 

Yes, it gives you a better understanding of weather. 

Of course! Because it was awesome to see the world and Mars fully. Also, how the land masses 
formed. 

Yes, it's educational and adds to the center. 

Yes, it's very educational and you can learn so much. 

\ es, it gives a firsthand view of what the earth and solar system is like. Wonderful Exhibit! 

Yes, our narrator explained things very well and the sphere can cover many topics. 1 especially 
liked that it had pictures of million years ago. 

Yes, you can see where your home was millions of years ago. 
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Yes, il gives you a moving 3-D look at our planet and others. 

Yes, because it was very interesting and fascinating for someone my age. 

Yes. It is very knowledgeable. 

Yes. It is interesting and fim. 

Yes, I would. It is very fun and educational at the same time. 

Yes, because you learn a lot. 

Yes. Because it's cool. 

Yes, because it explains a lot of things. 

My son (12) was feeling bored when we were going through some of the rest of the museum. He 
didn't want to come today. He sat through the sphere exhibit spellbound. 

Additional Comments 

El Nino was very good. Illustrative. Weather topics. The thing on global warming was good. I 
didn't know that sun's rotation was different. Good to see earth. Mars, variety perspective - Mars 
vs. Earth. Second time he's watching this, came yesterday. Remembered everything he said 
yesterday. Weather - El Nino? Understanding how hurricane on storm can come in, bigger 
storm before it came on shore (changes in storm) how it happens. Had always heard of El Nino. 
This is the first good visual I've seen. Had general idea that it was tropical storm that effects our 
weather. Now I understand how water warming takes place and how it effects us. The SO's one 
vs. the 70's one. I really liked it very much. 

At first you think the globe is moving. But it's not. It's definitely different. The only down side 
is that you have to keep moving. Good not to have distortion. 

Interested in renting it for a student physics circus in Texas. We have inner city kids - possibility 
for using it - Outside of their world - help them to see beyond Waco. Someone would have to 
work the crowd. Need interactive show. This is just a beautiful tool. 1 love that its not flat. Its 
like the globe on steroids. Grant writer for school; Husband is physics teacher at Baylor. You 
could do a GPS data project. Work with HS through grad. Students in science programs. Big 
outreach program. When he held up the nickel, my daughter said, “Think how small we are.” 
Inner city kids don't have perspective. They love anything that shows motion and time. There's a 
global information system curriculum in the technical college. To hook this up to geological and 
cultural systems data would be cool, in global information system, you can show whole 
civilizations growing. Could show it on this. Could combine this with global positioning system 
activities. Kids would make a cognitive leap. 

There is so much there to learn. So much of presentation you could focus on - Mars, Sun, 
history' of earth. The hard to believe theories of billions of years ago. Not sure how that helps us 
understand the earth today. Watching weather over AZ vs. other locations. Can see why AZ is 
desert. I would love to be able to take a group of 7th and 8th graders. You can see (he earth. 

This would stimulate kids. To be able to walk around, cool to be able to move around and see 
things. Like seeing formation of storms. In AZ - we don't have weather. Will take my students 
there if AZ gets one. Color - so engaging. When you see weather on TV you’ve focused in on 
one portion - here you see globe. From IT perspective - it's cool. Could talk about satellite 
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tracks, orbits of satellites, what’s where. - Former aerospace engineer - next year, will be high 
school teacher. 

I loved it, it's great. 

Absolutely incredible. 

I wasn't present for all of presentation. 

Sometimes you think you couldn't figure it out. 

Excellent. I enjoyed it greatly, too. I'm a science teacher. 
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ATTACHMENT I* 
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NUMMARY 
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i V* -Ci «iU h> m bvnjj -£*i -rjcfrfrJ m MrirA ym bcramexTwfrva' ■liiV.pnvAU l ^b'MU i |L 

-JifiMHpjr. 

liV ftfx A.' karr o p?i*- l^«|tar>nl^nn , i^t ikv.tViii^Inn ^dUiuu) .wi aMi ^ 

VunL-artvi on ■ol'i hSr k jU repV ■" 
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"Don'E h*ut CWlulaun %>. '.TijrftPMCfcm I «n a. rttm-ap.jin hi ■inn .Uiwrm 
• M fHff fid gMfiVnlflfiflh CfitAnEtiTK *in««- Tcrf ~'t.;j!:>-i.Iv. Uku llwj jjp tayn^j 
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blnw bkii. lallut 

If 

SL K 

w 

u 

M 


1 -3 

1-5 

1-4 

a-> 

1-5 

Warn T-Hr-j # 

z* 

4ft 

y.4 

39 

aa 

uunlFri' 

1 -4 

]-! 

7-5 

3-1 

1-3 

n ■! dnrgjiin 

?? 

35 

>J 

37 

» 

romaiunnlT 1 

1 -5 

I -1 

1-! 

3.5 

2.3 

43E«K*>MUnn 

5* 

37 

1.3 

3 N 

2.3 


1-4 

1-5 

3-3 

2-S 

B-S 

On fwitelr :,'>j rs 

34 

33 

IS 

3? 

J.3 


1 -t 

1 -T 

1 3 

2-S 

2-5 


U 

25 

?2 

3£ 

24 


1 -1 

] — j 

Z ' 1 

2.5 

1-4 


IJiViV liEw.imjT aUAin ilv r, mL trdk no} iL- giAihri k |t-.l *ilt ikf #U jnmft llf 

ham.iir nvJgmipy ariim 4 ii a ■.tjjiLt i*j>Lhtu ji JAir LiLl'i _t I iml firiV Li,i I'ljrrtniPK 
TTir inritabihli' LVTr rtrdv, irk lyu Vir -rm.ru.YT mini^ arf.Tsry. Jhr.Vpu firm ?** 1 
Hire' .Viv 4 *> Kn+irmtlu. 

Irmui utia i taniikxnkiHr |wp a s 
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ttww x-y; y. EiHHEi! Lmnptm 


'■'iintTTTimn 

1 ft™ ™i™ » l|UM rtlfl *uvsln^ -J|K>|. iilr. FiAmim, inj umn 

■■ priKTrf nun J film ri«nlipk,-iH. In ln.ni (if |1 k h m »ln.H •kilttun 

Ilf ■ inril-rj :ir.( ^ijpulHiuni. 

* to in yvtt okh- 

Mm fifr— . Mid L«J flirH; K£ ■■■■I'l' ffA fcili.il 
Li - 14 pw ni*: 

: . i p - lu tr.~- nvnj bJ kv I 4^r wi 1 1 -1 pi ci : h» ■ r’jh in I 

TfSthtl' 

I hvtlrn if m (Kin* « Tfwy «K». ftf-l kB^Vh| KunJn. ndk uhiic. ill id 
Ihj IkvI^i (Hiftn— tnlll imrrtrt'ifi 
■Y:-uld rjilp*- ni' i 1 ei dmujur iii'Ti-.'-i .'J 

■_ rv.i li » m.-ry iv-ai H-Hir-p hi Lidi h* .■ -Tr ■ iirwnrrvd m Hipir inBgfiiil(ia& Tfrfy 
t"/vi>n r Mn- -. '.i'll mid I . L- m lip- JiUm-nl n I h i>i dnctnr, pii r-m I rv->^i pj_ 
MtktduliA rtt.. — "Nllf line lipfc uf Hxnr. 

Add ■‘lUiil 

Sel mnhi vi n ■e'-Ti rti Frb 1 

Dun' I Lf-Ji r 4 jr J nr I a I ion 

FMk ffc. 

hint Wil About Ihv ll il Will loefc iv jl 'rd r_ ^.v. If Jur*nnd I'ajk tmj. 

( can’s mmtv peruw whaf IU4. ns-*i|,1 bf llkf ■ mUJll b^rtiLrt d^n2pnalrto!}wTr 
Ihc dirauicr ikvJwl rut metitt than ihr- . 4 c# Irion 
Han£ l.i pLi.tu rv 

DwfE p.+ Mil i y lJi> > jmrwjrv Ili rahihL LMor ISO trillion wjn >#:■, ■ r-.> warp 

mfJVr VfliWIHfa t-J.U +1 h&s fci-ui jtff JKu'rtfri iiksfr L- htlt« Huw dim t 

ww miK tetiue- H tilil tTirf 

l. L r-n c lirjii r-i.'i^il dir rrtaliiTiahp 

7Tr*r ^"ti vhtf I*.t rvtrp rair - im-,r iV uvil hr isiwim 1 ft notr a dar rmanihia 

EPIHUiA aid \W aSHfel. 

I im« Wlgrt_ 

Lbiift ynr fn^rr Id turn j inull whorl. v-ju can -ran I ml i num nn Ihp luhiiy ■'■ I Hi* 
«il 4 . Ift* iV-vLct ve"j Iujii, the luilcr lime REfc* by. iTinpjhr-rvIlTd imluM Min- up 
ih# I «t s>j4 «*-i i lire: d ••* ■■•--* ■*** Uht l' , . , ui »3i Grr luma uf iIh - wkwL In mam- like znlv a 
Imun ill riir mm 

tl — 3IJ 1.1UT ilk3 

r-pV-pk' ppm 't fjil r--r lun 

Duirt kiht in irredubvai- 

11 - irjr.i 4 d« 

Nik ihnl mlrmJii^ lor vc-iirujrr mill n n 
Hit irfeT Tv trrf 1 : inr«ftrnmr 

T>j« an ltt tv ii?-. N e a 1 1 iu I ■< and he^i - bnj!. ih- 1 1 1 n >.i . n. nm? Hiw 
^WChCT !iii m k may t» - l-jri i ije. 


Ik-. -J . m IJ^-n H. li rifc' 


r\w ib 
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Twtvn: 

£ ** w -"" 1 IwlHt vM? itffatbfr ii^' 1 mm luii mil hi be toukI ■ M.ifc1v 

kiiia ihruiJn * hi ahk» In mew Ifrf 

TIk L&icrtLvc beromes nirnni j wAwi li* rrtinxiy If ft* mbxi fru»ti hxr Luifi- 
«>ifihf cU» iirfWiiuIitii 
Sam 1 1 Ly: iif,. w[| Iwfl 

Puvnlx 

Wundarr il thu wuid i rmrr^s yci ir^rr )*H 7 - K. Ffid iuii V l-, wmid lk* 
wnALif. 

PTCfW lii'r-jj d=K3‘3c4ji)n _4 tnirlirr- fniflrf wrrt r-.llrr I |a^ irrH 1 .-j.jp,. | IB . J.«.. BJ | 
a IriTUflF G«Jthi*jfi4i!. 1 l 
Lten> Ilk'. L>rtr, l j|.rw wjjtfi "fy..Hii>. l ■Mivj^mii.iI 
Dot * librntatann 

!>?t? rFftMiftninx htw Itnap; nrrtain ■irtrarmcih hard Imply ^wlsiilii# ? b:nirl , -jii4 pin 
Miarhjri^ri rk'i! riltttiVc luc Ji.'vrnj,; time Epm 
f-p.rnir^ whrt?l np r .>.r Ul ;||y bjrfc.duw Itul- Kdbrk hour 

td^rtfKpr"? k^ l ■*p% , lAr JSrrwjrr. /fayjbli 1 * f-i-.-uu/Jw Cv i v . 1 is-J#rjc <i .l’.-i 
W l^al l/nrrpn v. 

hmkltttiJ Ihi 

Enlrr ■ ■■ n lilad *m mm md bfcnn* a [uLmimlrgi^ u-Jhiiinp; far ctart ibwil 
■Ji" L r^. Jr . a nd their mvirnttninl in rac k palir ms. iM«r !»a lo^ik *hf C-nte ftf 
■Ij.iiVj i.| fa fl q pf ii m. and burs->. Shurl Mslm- al K-ini; tv Id nwj.r.-tvip ih.'.rr 
haw rn bwi. ■WfliHR'. wJ k+c I>>jIi h> find >ial i feu tie ihr tsnuw lhal khv IkiM 
Kim. Link al petarH In IMd fulrtM hi .;...* t < 1 ir T ji^p firdinpc* In ml ipucianai. 

ml nnr.an- Jm. tu*r jm-u> kj (wren luYuh i jpaV tfi'Isc-iirK CAnV Lfcr.feU gnaJr 
■ Vi ftot-! 11 1 1 (f die Ienrt kerc [Srnr L*ir mr iVtx"ui r au' i tur .1 h*-j ll 1 ■ "’..I ik^w 

~m w ,tau* 

.Vtrtr rukrnjfi af'amiWKT cLmi mi' f.iiiu jk-xtk Qfri iVar/Fn'd nanlr tv-m&nn y but 
■I'H- nya ft 1 aYkt i i -. n ti nun* Is nVirrn, n*nl ,-iAhJi o.-li ■.>.-« i -is.-, s lsrr| h -,- |!, r | iVk .tf 
ddfWMtrl' mT rUtp PncK ijf? flva iViytI 

Cmm * mil a liajfi iii.-J.,! kAViN FHfkAW-n ntft i*i> earner di hvLH mdjM ctu 

l^uvosl in ■ .wainEHi tt .t iVais'-rJL-'. 

H l(? r nr i^da 

UkCHfl H>C w*r |p£d^j.' Ialk«->1 t> th«- vuJm 1^1 
W«T't wwf» tmanf |U« rifthc. 4 lillk >il'r (2 0 


■WiPWWls Pvkvi 


Kjs j II 
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11 -lltHTtUS 

VidrtM » lend ci P h'-lrw Lilli- row toadraii 

-jltEfti li-huli?* *nri hh F irfl wt-ai do im] wtuilhiijp vwsrr aka 

V-cy hjjri-un, achniMBHi ?au - vi- *.tt or-rcpf#d h* koff pftifefc ■. i taw 

E^b kid hinkflv “’ll mlnvyliryt 

R ■■■■'I y lik- Uib-«r<R dutbd I1R liani-m irtinlw 

I V-lin^y wuirt ■ cl niv+yr dfe^K- 
lucfacnc 

Mijchl hvM paid Kind pr.Hjp n-tmly - r.vb«*r ihjr. trm+v&j/A yrtnAJirt 

■J|Auv iQuilkiri Lu^; LTii>j|j;h, bul did Lila? ihf lul .’Hfwi lit rhb put 

Srjwr^f lie'll -j> k«Js ni' 1 1 rw+d 1 Ur miwurTi -hi kies -jvi- hi fjr. ^.iri i>r Jir.1 
t*tr a rlnsw Ir.Ti; Ji d jfttf-.nl p*"b irf Ur rauLil. - iraild hdw In r-r 

dilTnvrt I nr iw.- Jip.r- Irvri: pnmarf. mddk Ifegh vAu «L -rtc Fid pi puvrii 
fc kjK«, iihul itry ihoufcL hn it* kl.1i hi Jr. COuMiiw kxk j pr.m 

fcr filknp: wL I Ik hunt, muli tvsyll.i-; n»|Y- iH^MOd Hi Ur 

niu^ijln aid i»i€ U i ii' ^ i ns ifimniJ Uir TOprr>. m lh-> rrw wirn Ti-.Wif-j. 

CTOld -Pi y± .hmd Out pnJbCVun Uir bux Hul dirrv h ■ dnwndiir,, || pw-ihIj hi 
hf wry ihiiii^hl a>i1 arf OMpOfHl. hidi nmii lu lum oxthc N|» 
picturr xi mtII xa ana II pfrhjp? . 

Diirrmi I Jir *pprvf*nilr xraittPA Rilh ji thr «w ror iptfirfK port Uk ujer 
k-rcl? LWiujciaur Lidur-rudi ni. t - ■.■■.vr.rr ini pin-nli .;an IdffHify ijvtM 
■'r-p^'lv ftftHf kr Umr hiUb bul Lfo.ii fiain r u lm|r. ■§ Hi# pv^wAJir.- RIW 
rail wiri ii d-x-jld brukjp. aid tut> n-urh iwphfc riilhrlb#- 

L^hiiaL Itdi mipjil h# .■ll.-n-itf fcfji .-.£#■ HgrcgllH'i- ajiJ ddn fodi mifhl -ni 
■in y---u r^rr i.lmiiri hfc-.Ti*# r^r-, 1 Hunt cyy Af-f fx. T^lbjiih' 

ECirrmraiuanmi 'Ji-vakcp ■ cunsiilun iar utut lt«*y ^h^ijld k 4 fH IxiVcr. 

dufri't ,r , J -illn Uh.it (.■’jiTKn.Tcr Ux- rn_>riir- I L*y m_lrf rtrirJi TUhthfi^^ 
jck.TO.<k: 

Uenri *h. -.I.-J. m IHa j I m - irc.vj.T.r^j wi#. -ImI thc^ lums- hxI kxrrj-d. 
tjrirp Jif iW d’** — li* - 

I Jj«c- !-?.■ Ikjr-di us - a-.-u id h# pond IrrAhor! jitmHm n -. - j|i#b£- bini cik 

Uiii^g fcmufuBw 

I jfcf-h^v 1 1 fhcwf difkiCul k-triiiiK rifflci 

mwrinf ssmi.v Aid il ru my riti-ctivv uiri 

■i*rry will . 

■i^Jlon lu £ ±ts.Ttu! feMr-sl' kuming. 

0*34-1 liJrpjrfldT.- liLllk'. Hprill d 

Jw^iafi |jmp j. lli-liy- VpA'i'L Lxk.u Liirvu In •a-b‘x+ 1 . r!Tnp.uT. rt? 

InlFTKlKV IT. B nil JI1 i-|Mm1 

T'imfc: 

I &-£ k-a-iBHK prtKL'Mi BaLTitufr- iM'tCmbnud;.! 

>: MI. I.'f, lnk>4k«l^ii i.-^|-.- , .i Jlv P-X-J kw lilUr ;.:i+. 

CiKsd lor y.^ir^jAr and . J.Jer kifc. IrbCd-ilil m leantxip; Ik-v,- Rimtx-Jfl Irxm. Ukn 
i t aw -MB. irin a Birirlm 

Lil* :- h-‘ 1 1 1 ^ | ; aid rur-bmp, ikunicp. hnw %.i.-nli^>p.snL cnlu BAu.'itlLXZ. 

IPjtftallg lI's’K- 

ftifT# DUi LiivrciH jui'jfail .m.y ft* wip «dfei Jn iVfcrdjj;. 


LUiai h j-.rs:«. P. p. n.i 
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^JiuhlrJ LA 

I uiri j IdLiHiluc:^ “hcnr rrv:+ 1 of Lhc inpuiUiit inlmmuban jhiH dlncuun 

k dLA-Tnff#d firm fi!4d<0d ■X'llttWiJ ■* JM fidU. EuHnd .rating lhc iib in 
akilHbiru hwn 4l.lartnl Mlmali j*i 4 nyl UHpl<H I.'OlAjII l-iti lhc w’jrli. bki-fc*n 
aaj » dinmx’jrc hanp =fi ibr wilU 

LA-r kfAKUi IWb Sh_^ iC- in urdnll Is d^j; ^iauir,iniib .«j! of w. 

vl rattl'd UV-u^riii-i^jSiJ- jbfafoH fy> ttnMnvu foorftowft A> d* ibvmri urVlai. 

Uu*a rurrihiiir pLl.- b uV.inhi nv iL-jhi> i? riu'.iit wxu> ey 

I!- Wprt/L*j*i— 

LI - 34 1 1 if idda; 

Hi rtf* Mu- n.^4k' liLr- ihi 

Mos! kad* wjrt In dn karJi-m jrevu.w wlwi iNv mu-rur* *> ml ».u 

dfrMUnd. 

l , -«>»iri.4 > *tL*Xt -i d i Ktotomt t-kb^-Urn d:«->ii! pi ftsnnp, 

Tindian 

iV-frcnl'.’iii ■ ml -I ik dinvuur to mn.l.m. I Rtf frfiftfpT a diiMJisuirJ 
PJp.— oflrcmil li< ■ -null b_i rv 

Kid 1 !- ltr*I li'i.'Vk ijj iJij dij.k. iii:I u ■ r j; an -Lr-dnll waub fc* j r . -at r»»w 
■f^P 1 r - . iiii-, pnd Vtrp in I lt*A m wdi m ■_■■! itj S-.-n i ^ 

Sh=4A r, k-.- I.L.- M -LV- ttK. f»i>i1 ijnni >»«.«■■ ivhd? u.ndd il l«*c Ilk# wirti 
*■ ■- tjn '* - u* ria:wr,Til msipulwi ho*- - iC lui ivraph. 

SaIlIv Hina miv tv pn±ii#niiirL-#-rp uir dnJi;> 

Owd tv i IikIi luhJul ilwlcrii 
it.wrlr^liil IfriVKUcB 

Praia: 

'■WnUi' Irk- klh kkdrtib’i| u Inr^j m-c H <m jJI 
I ti^XTvifcp.Ti r.--; * 1 1 r« tJ 3vmrvor fior apcviaJ UftfaJ 

I br.5-i-.fi 

Lila KIITIIiIbT p«r. . -. -.1 In geOTJl 
Sramhricd Lit? vundi mol 

JYflflfr !*cH t*V |£y rin* Isml- iti kutkt mJ u-r il rimii m i¥w JWt cri n.T. *: 


i Ha it ill- 


fsn r> 
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UaUfiBTOPn lake lJ nr ri 4 .Uuj Y n* T 

Sft IN * diriManr nnL W*&db bkurt >i->ira an the hihiU nl dillrcpnl jrimalv, l!L* 
jji j L rv bird* *MP iunft'k Whkft lay rjg&si ar*-d uir I’ur li-irar ^un.i m dilffroi; hniy*. 
11 m lialan u|md M r.iiiK tv$ I.M birhpr (hey brlirs-r Jiaumn did ar did net laka 
‘.jrf nl lhair voun-p.. 


B LGTHriid*: 

PilfVjf'IK'dl lOnurdv iik itf xuu- bmHIrd hinny 
11 - M year rikla: 

Due lafa^iih- IwHpt Ixldikl /i - 1 1 nK-n^i^ 

Can ? krap Hneom uHim.:n Imp rn^i^h W.nrri ( o:- be real If Jltifi 
DW'tmnl UjidC watdh Ac hi dm- rwd an ■rtmtj ■».vvTWTipinr pt 
‘.■ iTpHJ'anK In lr«»\ 1 1 an'»j 14 1 1 in irik*r k' av llw ha hi at- rrvMT har.la-rn dun ‘■Ifgd 
All ppw# 11 rttrt-yl vum-taid? «i wivjtw 

T-nchara 

Jlt liiih xhidpnr-. I K.' r. ■ 1 1 pj n %zifi n fj- _ i ■: 

Tlp-ui .4 f'fi.m 1 1 LT.\*i- - i>tLTns and hnw dillanral jramah r# I hr 1 ppmaS I 1 i-d >. j-aaj j. ■ 
KC befeV iJira^ji -. ifO.-a<.lnl wtfh -.tiler ‘cinn 
{Jirtc I k-iTaT .V. fliw: tjf the irfa.iiifrx'' 

: -, -k.v-±. .ari iurmuhhi pmrnalnj 


Parcntl! 

H>ylW hi*-* Idu-jl IWHS 

Jilhnhrn ^jun nl .rhjMrrfi ITblJ l <■ KM tfwrt. Apprdi rrm In r.rfc. 

Lil^ ihr mrjiipl bul tno mii;h uMchir£ RdS Wifi SpI ihnnqji il 
QK hi- ht -pj urJs.T- pran: nl anjedivU 

P-^jpk 1 tvrf wtfJ iAvvt kw*.tir*B red® in itv Aivuir u-j. Tfey p f^r 

kanda-aBAjfrnlbn 


Wll T.fil?. ■ Mm In fcillfrfBf V*rraih c.ninr>tarf 

I'M 1 a -ii a 11 k hi inure a lQ-luol haah sin-lei nl a nrr±anii-> ji'l-. |pg Jml Iriini kju. Rime 
dhPOHUrlffd Piifthl lire kn bnill be nun dr. IValth vid#*a of animal? HVk<- 
MhuhH jbJ plpfhini-j rupplflft Ji-a wn pa ravin. I.bhcn hi two an an baba njdiln wly 
■Ihrr linisk a^anrf dlncuum - liil.l nr ratiM pal ChA u|> 10 411 miln a hour In ralrh nn7T. 

4 - 10 yuan old ■ — 

II- lly*inn?kl 

IJkJ- I na|l I" 1 1 Fp " --T. r *ji rr ijiit w.iriid p»ir+_ ,VJow lha: il Uinpljan^d umuJ 

ml. hpl-.T sun .1 i K -r-y?l 

TioJktk 

!»*■ ll’ifip- ‘.^■ETrfA - and uhi Kianhhc indind 


Rnmtx 

herd !■.'■ rjiY 1 uim ■“in 

i.VMfpjIrr 1 1 1 ■ ■ 1 a 1 1 l U Tidoaa 

NWv ftMS in Wrindi Fun'ii net anl vkLv;- 


ri-ii-ii him ihr Ha 1 b i.uii r- hhiki 
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BM Dina j bps penny q]«lcr 

iurninv Lfcr hcUau -mil anUpnli dmrvutrV. h#jd r>»p#r1nif hi w«fc ah l.w lu 
Hike HiiaJi Ihr >tiran uJ dtrlriminp al Uw ilinr-jur ru} Vjmp ■■tail il«"+. 

B 4I3HB ni >. i hr "i4 «4brc dinoMon 


1 - prom rJ .1 — 


I L - H *!inn lid: 

Friiika! wmdkfl rv rrlrmdinp In m li :■■.*. llw riimumn rrmrriinrj«i>1 - wn r thfj 

lud m scy biMdidv 

Trtfc-fwri: — 


F'HCiii: 

Huw tin Lite# fcv ilcnr? 


WJtriJSlKiffl On TndstTdA SKT 

rc-iluH’ tfcjp iDfyktfd harlot -od twa Maryland dianusn ik# mtnpiron-. plinl-rMu^f 
, J iblpjckiii and dir I'mciim. air jI-i j'iRP Ajctw anlhcuum. WjD urJ.v iti^ fu|| -§£||t 
*?f Ihor fm -iauiuan *a- Ihr /ivirKanlttciiBUFji iiiirk;- Ihr bijjirr, bur itawn 
AalrvHlin ihr fhidMCt Of ihf Inthi ind ihr Inlf id fine lt»t* dinouun u- 

lindi-Hil wKil #ranl ail phi hav# uli-i p|iva- 

3 - tii'-jFran- old: ■ — 

n - n retrfcijji 

Mewp pwuil to lx>5^- VW*C fwe nifih t ie »,;■■. .>>kn f>i hv lu ASM pifc 
Tevhcn: 

■E4k“ r - 1 f ~ L" irx - Li.r. k- in: udr bjp :-iik ndwr-a-jai "_jv r cr nncdi im IK? h-arts. 

hit iJ Jp0u link*.- il tftirr trf a H.avtnp.T runl ih.-j'l Kan x'-d: mnrv ihiuir^. 
morf- rvutlaJfr A toftffcp pfOMAk HUfl£ SI Ac U.r.k- firal aril hv^-h Ahp 

+.~. rnh-x- nli rr-r-r*i rturrri. tfv-ri ui|ffpf( 

Lfcc tki h.™u? il »i mywtwrj mu bjv H£t.rvtg nwlhthfiiML Shim lim 
dinHounh ittiuiv mlrrelLii 

Kjfclll i'Jri. OWittid i" ktov HupccC &tpa rrfi- lr.i-.v_-i I p.-iti t It-# mr-ilrla !hnw InsfL^ 
iirrJ. ha Vi- tt.rf.-f.b u-tlijr kind uf din mi- ihcr. k-ad Ihrn adz i j ni-r-rn 

with ihr ni.--.li -la 

Tarcrtr; 

LoaJ ifltaut 

Snw nc« MD tTJdtf h-TnCltfu'i frxj-iJ f. t«]J brhxrd lur pnplr bs kn:i- <i*ak Ik?. 
II «|w-ruljll:n. nnt* IWUHl u-WtaHTI b Murugud bul rid! rr.vLV-*irib bm 
hi iilm. Dai'l hirer UL^rciuvrliivi .r rtiinir- 
h'udj jiw l;i krrr j MDimmictKn- wart In kmm Jh-tnal Mil iivvr iin .^i 
map "-L 1 fill'd hum p/dllv wIic-tv Urv JulY >■!* k-urd. W jJrr vip^m and 
ihlaEtn ‘■'Cf> p Sirr^'toik tell *ja ii dw pill r kw ■ I!-. - - Iv.-C nil diuKtur, 
nf* a Kk^a-Wi [III r.LiL*v 1 ?r.v|rii'fTi?"i 
uoud l"rr irhidp.'" In uv- fna^ AaJifu 
Lhdn’l rr.ai i Mr UD enindiir 


ivi* i i 



162 


MifrDidW— n n Ink. iJin- 1 

Chi A ttMtiiaAri. Ai/hw vam-t-wa dir.-niam jt.j add ai jlH nr ir-jil>-iv u hilly h*sir yrt Is 
b+ Inaand wAfh tond I. 1 Find mt ishy iJilr-irrrnl diuuur am -da b4j> d*. hVcriil ■. uVi r 
and marking!. i& palnl .llm>u*ir.- bid Mkrt lIwv Ihry pq-rl Inin livnfl aal null 

P 14 vv ii ra uld. 

Ukjb art 


n - U ywvoH 
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ATTACHMENT C 
Dinosaur Mysteries 
Summative Evaluation 


Minda Borun, Museum Solutions 
February 2005 


Introduction 


Summative Evaluation of the Dinosaur Mysteries Exhibit at the Maryland Science 
Center was conducted from November 2004 through January 2005. The study has two 
main parts: 1) tracking and timing of visitors in the gallery and "sweeps" or counts of 
the number of visitors at each exhibit station and 2) exit interviews of visitors leaving 
the gallery. Tracking and timing and sweeps give a measure of the relative attracting 
and holding power of exhibit components. They tell us where people go and how long 
they stay, which components are compelling and which components are missed. Exit 
interviews give visitors reactions to the exhibit as a whole and to its components. They 
tell us what people like, what they learn and how they interpret the exhibition. 

This report is based on 97 tracking and timing observations and 95 sweeps conducted 
between November 4 and December 28, 2004 and 80 exit interviews conducted on 
January 8 and 9, 2005, 
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Executive Summary 

• Visitors spent an average of 17 minutes in the Dinosaur Mysteries exhibit and 
stopped at 9 exhibit components. 

• The top scorers in terms of attracting and holding power are primarily 
interactives. 

• Visitors associated the exhibit with a list of subjects 

• Visitors were aware of many of the activities that put them in the role of scientist. 

• Visitors could list issues concerning dinosaurs about which scientists disagree. 

• Most of the visitors interviewed were able to cite ways that the earth had changed 
over time. 

• Visitors were able to give several sources of evidence for dinosaurs having been in 
Maryland. 

• Most visitors were aware of several theories of dinosaur extinction A few visitors 
made up their own extinction theories l 

• Most people came upon the exhibit while exploring the museum. Some had heard 
about it from somebody. Few had seen marketing message about the exhibit. This 
is typically the case with visitors to special exhibits. 

• More than three quarters of the subjects had seen other dinosaur exhibits. 

• Visitors had seen dino exhibits in a wide variety of places, but the Smithsonian 
was by far the most frequently cited. 

• When asked for their favorite exhibit, the list is a little different from the tracking 
and timing and sweeps data since it includes live animals in the top favorites. 

• Only 15% of the respondents had problems with some of the interactives. 

• Visitors gave the exhibit very high ratings. 

• Almost all of the respondents said they would return. 

• Visitors to this exhibit seem to be emphasizing quality rather than quantity. 
While the average visitor sees only 9 out of 67 exhibit components, they seem to 
understand the exhibit content and are able to explain the main messages. 
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Tracking and Timing 


• Visitors spent an average of 17 minutes in the Dinosaur Mysteries exhibit. Times 
ranged from 1 minute to 45 minutes. Visitors stopped at an average of 9 exhibit 
components. 



Mean 

Range* 

Mean % 

Length of Visit 

16 min. 57 sec. 

1 min - 45 min. 32 sec. 


N of Elements (73) 

9 

1-28 

13% 


^Maximum time in Dinosaur Mysteries was determined by subtracting time in TerraLink from total time 


A ge /exhibit considerations 

The holding time of some families may be shorter than expected because of they included 
young children. Over half (39) of the 68 adults that were tracked were accompanied by 
children 7 years old or younger. 


As is usually the case in science museums, the interactives held people’s attention for the longest 
times, although the dramatic Parasaurolophus skull is also in the top 10. On the other hand, text 
panels tend to cluster at the bottom of the list. 


Table 1: Time in s econds at exhibit com ponents (N-97) 


Rank 

Map # 

Exhibit 

Mean 

Range 

1 

F6 

Color a dinosaur (2) 

302.2 

28 - 756 

2 

Ell 

Create a Dinosaur (2) 

266.4 

30 - 880 

3 

B2 

Dig activity 

233.7 

5-1223* 

4 

F5 

Coloring 

232.8 

17 - 635 

5 

A3 

Continents on the Move (2) 

151.7 

26 - 581 

6 

A4 

Timeline spin browser (2) 

150.7 

6-493 

7 

E8 

Parasaurolophus skull 

125.0 

125-125 

8 

A7 

Is it a dinosaur? 

117.8 

10 - 386 

9 

Cl 

Air scribe (3) 

110.2 

10-588 

10 

E3 

T. Rex debate 

99.0 

13-338 

11 

C4 

Puzzle 

87.0 

6-440 

12 

B8 

Claws 

84.7 

16 - 197 

13 

C5 

Juvenile Albertosaurus 

84.5 

9-314 

14 

B12 

Measure tracks 

78.2 

9-395 

15 

F7 

Trackway Debate- video 

75.4 

10-285 

16 

G4 

The Bird Debate- video 

70.0 

9-178 

17 

E10 

Hollow Crest Debate- video 

68.5 

2-330 

18 

B7 

Teeth 

67.5 

8-103 

19 

F8 

Making tracks 

67.1 

5-260 

20 

Bll 

Identify plants field station 

64.8 

58-83 
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Tracking continued 


Rank 

Map# 

Exhibit 

Mean 

Range 

21 

B3 

Measure a bone 

63.2 

8-144 

22 

E9 

Make a Dinosaur Song 

62.6 

3-156 

23 

B9 

Feet and legs 

60.2 

11-112 

24 

E14 

The Parenting Debate - video 

59.9 

7-285 

25 

B5 

Ammonite patterns 

59.7 

10-178 

26 

A6 

Monitor lizard, frogs, etc. 

54.2 

6-319 

27 

B6 

Rock patterns 

53.2 

8-146 

28 

E4 

T. rex skull and video 

53.0 

25-89 

29 

B13 

Mud 

52.3 

5-152 

30 

G1 

Core sample 

51.0 

51-51 

31 

BIO 

Introduction video 

50.0 

50-50 

32 

B1 

Introduction video 

48.0 

9-106 

33 

D4 

Astrodon femur 

46.4 

11-101 

34 

A1 

Gigantosaurus/Compsognathus 

41.6 

9-113 

35 

C6 

Vertebrae comparison 

39.4 

6-107 

36 

G5 

Archaeopteryx plaque 

38.8 

10-58 

37 

A 

"Dinosaur Time" (text) 

37.4 

4-221 

38 

B14 

"What is a fossil?" (text) 

36.8 

17-67 

39 

El 

Triceratops skull 

36.4 

5-313 

39 

FI 

Astrodon/ Arcocanthosaurus 

36.4 

2-139 

41 

E7 

Latest Greatest Finds- video 

35.7 

7-121 

42 

F.17 

Egg fossil 

35.6 

3-134 

43 

E2 

Misc. skeletons 

34.7 

5-76 

44 

E5 

T. rex model 

33.8 

10-76 

45 

F4 

Artist Brings Scenes to Life- v 

33.6 

8-80 

46 

G7 

Collections graphic 

33.0 

12-53 

47 

E13 

Oviraptor skeleton 

32.5 

9-63 

48 

A5 

Dinosaur Timeline 

32.3 

11-74 

49 

F 

text- Maryland Dinosaurs 

32.2 

4-98 

50 

F2 

Trackway model 

32.0 

17-71 

51 

B4 

Identify animals 

31.6 

8-68 

52 

C2 

Fossil in rock 

31.4 

9-110 

53 

E15 

Climb in nest 

30.7 

8-111 

54 

D2 

Emmetsburg tracks 

30.4 

9-56 

55 

G3 

Bird and dinosaur skeletons 

29.0 

29-29 

56 

A8 

Tarbosaurus skeleton 

28.9 

6-71 

57 

E12 

Dinosaur blocks 

28.5 

8-83 

58 

E6 

Herrerasaurus skeleton 

27.9 

6-43 

59 

D3 

Positive negative tracks 

27.6 

6-103 

60 

F3 

Process artifacts 

26.3 

9-67 
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Tracking continued 


Rank 

Map # | Exhibit 

Mean 

Range 

61 

C3 

Fossil vs. cast 

25.5 

12-38 

62 

D5 

Arcocanthosaurus teeth 

21.8 

8-36 

63 

D1 

Juvenile Astrodon 

21.0 

13-29 

64 

A2 

Globe 

20.5 

7-51 

65 

E16 

Compare nests 

18.1 

6-64 

66 

Ba 

"Finding Evidence" (text) 

16.5 

10-23 

67 

Bb 

"Finding Evidence" (text) 

15.0 

15-15 

67 

G6 

Caudipteryx plaque 

15.0 

15-15 

69 

C 

"Reading Evidence" (text) 

14.0 

14-14 

70 

AO 

Entry text panel 

10.3 

5-22 

71 

E 

"Dinosaur Mysteries" (text) 

0.0 

0 

71 

G 

"What Happened?" (text) 

0.0 

0 

71 

G2 

The Extinction Debate (video) 

0.0 

0 
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Sweeps 

Similar to the tracking and timing results, sweeps show that most people are clustering around 
interactives. 


Table 2: Visitors at each exhibit component * 


Rank 

Map # 

Exhibit 

Number 

Percent 

i 

B2 

Dig activity 

441 

13.0 

2 

Cl 

Air scribe (3) 

250 

7.4 

3 

C5 

Juvenile Albertosaurus 

247 

7.3 

4 

A4 

Timeline spin browser (2) 

162 

4.8 

5 

F6 

Color a dinosaur (2) 

148 

4.4 

6 

Ell 

Create a Dinosaur (2) 

138 

4.1 

7 

C4 

Puzzle 

131 

3.9 

8 

F5 

Coloring 

128 

3.8 

9 

A3 

Continents on the Move (2) 

119 

3.5 

10 

F8 

Making tracks 

103 

3.0 

11 

A6 

Monitor lizard, frogs, etc. 

97 

2.9 

12 

B12 

Measure tracks 

95 

2.8 

13 

A7 

Is it a dinosaur? 

71 

2.1 

14 

E15 

Climb in nest 

70 

2.1 

15 

FI 

Astrodon/ Arcocanthosaums 

69 

2.0 

16 

B13 

Mud 

62 

1.8 

17 

F7 

Trackway Debate (video) 

57 

1.7 

18 

E14 

The Parenting Debate (video) 

54 

1.6 

19 

E3 

T. Rex debate 

48 

1.4 

20 

B3 

Measure a bone 

43 

1.3 

21 

B5 

Ammonite patterns 

38 

1.1 

21 

E17 

Egg fossil 

38 

1.1 

23 

A5 

Dinosaur Timeline 

37 

1.1 

23 

E9 

Make a Dinosaur Song 

37 

1.1 

25 

F4 

"Artist Brings Scenes... "(video) 

35 

1.0 

26 

C6 

Vertebrae comparison 

34 

1.0 

27 

G2 

The Extinction Debate (video) 

32 

0.9 

28 

B4 

Identify animals 

30 

0.9 

28 

G4 

The Bird Debate (video) 

30 

0.9 

30 

E10 

Hollow Crest Debate (video) 

28 

0.8 

31 

A2 

Globe 

27 

0.8 

31 

El 

Triceratops skull 

27 

0.8 

33 

E4 

T. rex skull and video 

26 

0.8 

34 

E5 

T. rex model 

25 

0.7 

35 

B7 

Teeth 

23 

0.7 

35 

E7 

Latest Greatest Finds (video) 

23 

0.7 

37 

A1 

Gigantosaurus/Compsognathus 

21 

0.6 

37 

E16 

Compare nests 

21 

0.6 
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Sweeps continued 


Rank 

Map # 

Exhibit 

Number 

Percent 

37 

G7 

Collections graphic 

21 

0.6 

40 

B8 

Claws 

19 

0.6 

40 

G3 

Bird and dinosaur skeletons 

19 

0.6 

42 

F 

"Maryland Dinosaurs " (text) 

18 

0.5 

43 

B6 

Rock patterns 

17 

0.5 

44 

A8 

Tarbosaurus skeleton 

16 

0.5 

44 

B9 

Feet and legs 

16 

0.5 

44 

D3 

Positive negative tracks 

16 

0.5 

44 

E12 

Dinosaur blocks 

16 

0.5 

44 

F2 

Trackway model 

16 

0.5 

49 

A 

"Dinosaur Time" (text) 

15 

0.4 

49 

B1 

Introduction (video) 

15 

0.4 

51 

F3 

Process artifacts 

14 

0.4 

52 

C3 

Fossil vs. cast 

13 

0.4 

53 

C2 

Fossil in rock 

12 

0.4 

54 

G5 

Archaeopteryx plaque 

11 

0.3 

55 

G6 

Caudipteryx plaque 

10 

0.3 

56 

D2 

Emmetsburg tracks 

8 

0.2 

57 

D4 

Astrodon femur 

7 

0.2 

57 

E2 

Misc. skeletons 

7 

0.2 

57 

G1 

Core sample 

7 

0.2 

60 

BIO 

Introduction (video) 

5 

0.1 

60 

B14 

"What is a fossil?" (text) 

5 

0.1 

62 

D1 

Juvenile Astrodon 

4 

0.1 

62 

E13 

Oviraptor skeleton 

4 

0.1 

64 

Bll 

Identify plants field station 

3 

0.1 

64 

D5 

Arcocanthosaurus teeth 

3 

0.1 

64 

E6 

Herrerasaurus skeleton 

3 

0.1 

64 

G 

"What Happened?" (text) 

3 

0.1 

68 

AO 

Entry panel (text) 

2 

0.1 

69 

Bb 

"Finding Evidence" (text) 

1 

0.0 

69 

C 

"Reading Evidence" (text) 

1 

0.0 

71 

Ba 

"Finding Evidence" (text) 

0 

0.0 

71 

E 

"Dinosaur Mysteries" (text) 

0 

0.0 

71 

E8 

Parasaurolophus skull 

0 

0.0 


*Based on 95 sweeps totaling 3392 visitors 
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Score 

If we multiply the mean time (from tracking and timing) by the percent of total visitors 
at the exhibit component (from sweeps) we get a "score" for the components that 
combines attracting and holding power. Not surprisingly, the top scorers are 
interactives and the text panels fall to the bottom. The fact that the Parasaurolophus skull 
is also on the bottom is an artifact of the fact that there were no visitors looking at it when the 
sweeps counts were made. On the other hand, the Juvenile Albertosaurus had a pretty high 
score. This is why all three tables need to be considered. 


Table 3: Score for each exhibit component 


Rank 

Map # 

Exhibit 

Mean 

Percent 

Score 

1 

B2 

Dig activity 

233.7 

13.0 

3038 

2 

F6 

Color a dinosaur (2) 


4.4 

1318 

3 


Create a Dinosaur (2) 


4.1 


4 

F5 



3.8 

878 

5 

Cl 

Air scribe (3) 


7.4 

812 

6 

A4 

Timeline spin browser (2) 


4.8 


7 

C5 

Juvenile Albertosaurus 

84.5 

7.3 

615 

8 

A3 

Continents on the Move (2) 


3.5 

532 

9 

C4 

Puzzle 

87.0 

3.9 

336 

dm 

A 7 

Is it a dinosaur? 


2.1 

247 

K 1 


Measure tracks 

78.2 

2.8 

219 

El 

F8 

Making tracks 

67.1 

3.0 

204 

mm 

A6 

Monitor lizard, frogs, etc. 

54.2 

2.9 

155 

mm 

E3 

T. rex debate 

99.0 

1.4 

140 

mm, 

F7 

Trackway Debate (video) 


1.7 

127 

■a 


Mud 


1.8 

96 

mm. 


naiaMBM 


1.6 

95 

mm 

B3 



1.3 

80 

mm 

FI 

Astrodon/Arcocanthosaurus 


2.0 

74 


E9 

Make a Dinosaur Song 

62.6 

1.1 

68 

mm 

B5 

Ammonite patterns 

59.7 

1.1 

67 



Climb in nest 

30.7 

2.1 

63 


G4 

The Bird Debate (video) 

70.0 

0.9 

62 

mm 


Hollow Crest Debate (video) 

68.5 

0.8 

57 

mm, 

B8 

Claws 

84.7 

0.6 

47 

mm 

B7 

Teeth 

67.5 

0.7 

46 

' • 

E4 


53.0 

0.8 

41 

mm 

C6 


39.4 

1.0 

40 

mm 


Egg fossil 


1.1 

40 

141 

A5 

Dinosaur Timeline 


1.1 

35 

30 

F4 

"Artist Brings Scenes" (video) 

33.6 

1.0 

35 

32 

El 

Triceratops skull 

36.4 

0.8 

29 

33 

B4 

Identify animals 

31.6 

0.9 

28 

33 

B9 

Feet and legs 

60.2 

0.5 

28 
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Score continued 
Rank Map # 

35 B6 

36 A1 

37 E5 

38 E7 

39 B1 


43 A2 
43 G3 

45 F2 

46 A8 

47 D3 

47 E12 


50 E16 

50 F3 


Exhibit 

Rock patterns 

Gigantosaur us/Compsognathus 

T. Rex model 

Latest Greate st Finds (video) 

Introduction (video) 

Collec tions graphic 

"Din osaur Time* (text) 

"Maryland Dinosaurs" (text) 
Globe 

Bird and dinosaur skeletons 
Trackway model 
Tarbosaurus skeleton 

Positive negat ive tracks 

Dinosaur blocks 
Archaeopteryx plaque 
Fossil in rock 

Compare nests 

Process artifacts 


Mean | 

53.2 

41.6 

33.8 

35.7 

48.0 

33.0 

37.4 

32.2 ~ 

20.5 ~ 

29.0 

32.0 

28.9 

27.6 
28.5 

38.8 
31.4 ~ 

18.1 ~ 
.3 








Percent 

Score 

05 

27 

0.6 

26 

0.7 

25 

0.7 

24 

0.4 

21 

0.6 

20 

0.4 

17 

05 

17 

05 

16 

0.6 

16 

0.5 

15 

0.5 

14 

0.5 

13 

0.5 

13 

0.3 

13 

0.4 

11 

0.6 

11 

0.4 

11 
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Sample Description 


Table 4; Demographics of Visitors for Tracking and Timing 

Number Percent 


Adult Males 

30 

32 

Adult Females 

38 

39 

Boys 

14 

14 

Girls 

15 

15 

Totals 

97 

100 


Table 5: Demographics of Children who were tracked 


Number 

Number 

Totals 

Boys 6-10 

5 

Girls 6 - 10 

9 

14 

Boys 11 - 15 

8 

Girls 11-15 

3 

11 

Boys 16 - 20 

1 

Girls 16 - 20 

3 

4 

Totals 

14 

15 

29 
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Exit Interviews 


The following results are based on 80 interviews conducted on January 8 and 9, 2005 
When multiple visitors give the same answer, the number of visitors is shown in 
parentheses ( )! 

1. Where do you live? 

• Over seventy percent of respondents come from Maryland. 


Table 6: Residence 

Number Percent 


Baltimore 

16 

20% 

Other Maryland 

43 

54% 

Virginia 

7 

9% 

Pennsylvania 

4 

5% 

Delaware 

3 

4% 

New Jersey 

3 

4% 

New York 

2 

3% 

Washington DC 

1 

1% 

Lima, Peru 

1 

1% 


2. Your age? 

• The most frequent ages are 30-49. 

Table 7: Age 


Number Percent 


10-14 years old 

5 

6% 

15-19 

2 

3% 

20-29 

11 

14% 

30-39 

31 

39% 

40-49 

19 

24% 

50 - 59 

8 

10% 

Over 59 

4 

5% 


' Some percent totals do not add up to 100 due to rounding. 
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3. How many people are in your group? 

• People come in groups of 2-4 people. 

Table 8: Group Size 

Number Percent 

One 3 4% 

Two 25 31% 

Three 18 23% 

Four 23 29% 

Five 6 8% 

Six or more 5 6% 


• The respondent groups included slightly more adults than children. 

Table 9: Gender 

Number Percent 

Adult males 74 30% 

Adult females 65 26% 

Boys 64 26% 

Girls 47 19% 

Totals 250 100% 


4. With whom did you come to the Dinosaur Mysteries Exhibit? 

* Eighty percent of visitors came with family groups. 

Table 10: Group Type 

Number Percent 

Alone 3 4% 

Friends 8 10% 

Family 63 80% 

Family & Friends 5 6% 

Group 0 0% 
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5, What would you say the Dinosaur Mysteries Exhibit is about? 

( multiple answers were accepted ) 

• Visitors associated the exhibit with a list of subjects 

Paleontology, archeology, research on dinosaurs (25) 

Natural Science/ geology (15) 

Dinosaurs (12) Educational (3) 

How dinosaurs lived /behaved (9) Maps (2) 

Age of Dinosaurs (7) Live animals (1) 

Maryland dinosaurs (7) Electronics (1) 

Fossils (6) Tornado (1) 

History/ timeline of dinosaurs (6) Weather (1) 

Evolution of Dinosaurs (5) Volcanoes (1) 

Dinosaur extinction (3) Experiencing family (1) 

5. Dinosaur Mysteries puts you in the role of a scientist. Can you tell me some 
things you did in this exhibit that are what a scientist does? ( multiple answers 
accepted) 

• Visitors were aware of many of the activities that put them in the role of scientist . 


Activities at dig (50) 
Airscribe/drilling (16) 
Measuring (10) 
Archeology (5) 

Fossil hunting (4) 
Computer (4) 
Observation (4) 


Water globe (I) 
Time line (1) 
Coloring (1) 
Videos (1) 
Tracking (1) 
Compare teeth (1) 
Field notes (1) 


Identifying claws, tracks, foot prints (2) Puzzle (1) 

Eggs (2) Skeletons (1) 

Foot prints (2) Earth and climate (1) 

Separate rocks and bones (2) Making foot prints (I) 

Discover plate tectonics theory (1) Matching bones (1) 
Use magnifying glass (1 ) Examined fossils ( 1 ) 


Form hypothesis (1) Sit in nest (1) 

Examine rock strata (1 ) Put skeletons together (1 ) 


Deduce things based on evidence (1) Movies about extinction (1) 


Study craters of meteors (1) 

How they document, discover, date fossils and dinosaur habitats (1) 
Conjecture what they looked like/ what they ate (1) 


No answer (3) 
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6. What are some of the questions about dinosaurs that the experts still disagree 
about? (multiple answers accepted) 

• Visitors could list issues concerning dinosaurs about which scientists 
disagree. 

Why dinosaurs are extinct (18) 

Their color/appearance (11) 

Bird debate (8) 

Trackway debate (8) 

T-rex debate (7) 

How long ago they lived (4) 

Their behavior (2) 

Warm or cold blooded (2) 

Origins of dinosaurs (2) 

Bones (2) 

Relationships between families (2) 

7. What did this exhibit tell you about how the earth changed over time? ( multiple 
answers accepted) 

• Most of the visitors interviewed were able to cite ways that the earth had changed 
over time. 

Continents moved (28) Volcanoes (1) 

Climate changed (22) How earth became more stable (1) 

Pangea/ one continent (11) Different soil strata (1) 

Animals changed (9) Earth moved (1) 

Extinction of species (including dinosaurs) (8) 

Meteor/ asteroid/comet (6) 

Evolution (6) No answer (20) 

Changes in plants/ vegetation (3) 

Land changed (3) 

8. How do we know that there were dinosaurs in Maryland? 

(multiple answers accepted) 

• Visitors were able to give several sources of evidence for dinosaurs having been in 
Maryland. 


How they protected their eggs (1) 
Plates (1) 

Body part functions (1) 

Mammal transition (1) 

How they travel in packs (1) 
What they ate (1) 

Sounds made by hedredon (1) 

No answer (31) 


Fossils (46) 

Tracks/footprints (15) 

Bones (13) 

Work of paleontologists (1) 
Model of Maryland dinosaur (1) 


Saw label (1) 

Remains in rock strata (1) 
They were all over (1) 

No answer (14) 
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9. Why are there no more living dinosaurs? What happened to them? 

(multiple answers accepted) 


• Most visitors were aware of several theories of dinosaur extinction A few visitors 
made up their own extinction theories ! 


Meteor /asteroid /comet (38) 
Climate change (18) 

They are extinct (10) 

No one knows (7) 

Ice age (7) 

Volcanoes (7) 

Lack of food (6) 

Evolution (4) 

Still around as birds (4) 
Habitat change (3) 

Big Bang (3) 

Ate each other (3) 


Man (2) 

Died from disease (1) 
Couldn't migrate (1) 
Nuclear winter (1) 
Atmospheric pollution (1) 
Explosion (1) 

Tornado (1) 

Earthquakes (1) 

No answer (8) 


10. How did you find out about the Dinosaur Mysteries exhibit? 

• Most people came upon the exhibit while exploring the museum. Some had heard 
about it from somebody. Few had seen marketing message about the exhibit. This 
is typically the case with visitors to special exhibits. 


Table 11: How found the exhibit 

Number Percent 


Exploring the museum 

35 

44% 

Heard from someone 

12 

15% 

who had seen it 



Seen it before 

13 

16% 

Museum map 

0 

0% 

Internet 

5 

6% 

Other* 

15 

19% 


* Members (5), newspaper (4), word of mouth (1), hotel brochure (1), 
radio (1), billboard (1), media (1), previewing for scout trip (1) 

11. Have you seen any other Dinosaur exhibits? If "yes" how is this one different? 

• More than three quarters of the subjects had seen other dinosaur exhibits. 

Table 12: Saw other Dino Exhibits 

Number Percent 

Yes 63 79% 

No 17 21% 
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* Visitors had seen dino exhibits in a wide variety of places, but the Smithsonian 
was by far the most frequently cited. 


Other Dinosaur Exhibits 
Smithsonian (37) 

AMNH, NY (6) 

Field Museum, Chicago (3) 
Philadelphia (2) 

British Museum (2) 

Dallas, TX(1) 

Liberty Science Center, NJ (1) 

Sand's Point Museum, Long Island (1) 


California Tar Pits (1) 
Virginia (1) 

Alabama (1) 

Carnegie (1) 

Calgary, Canada (1) 
Atlanta airport (1) 
Dinosaurs Alive, Del, (1) 


Comments on differences 


* Most of the visitors who had seen other dinosaur exhibits said that this exhibit is 
more interactive 

This exhibit is more hands-on/interactive (30) 

compared to the Smithsonian, British Museum, Carnegie, Field, Calgary, 
Dallas and AMNH 

This exhibit has fewer hands-on than Liberty Science Center (1) 

This exhibit is more child-oriented (7) 
compared to Smithsonian and Field 
This exhibit is smaller (6) 

compared to AMNH, Smithsonian, Liberty Science Center 
This exhibit is more fun than Smithsonian (2) 

You learn more in here than at the Smithsonian (1) 

Virginia has no globe. There are more pictures and computers here. (1) 

There's a bit of everything here; not text heavy (1) 

The Smithsonian has no skin models, more fossils (1) 

I liked this one better than Smithsonian (1) 

The Field Museum has bigger bones. (1) 

The Smithsonian is free. It hasn't changed for years. This one is newer. (1) 

This is more practical- no real bones (1) 

This is more about atmosphere, more scientific and more for adults than 
Dinosaurs Alive (1) 

This is about dinosaurs in the future. The Smithsonian, about dinosaurs in the 
past (1) 

This is not animated: Sand's Point Museum on Long Island is better (1) 
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12. Which exhibit station(s) in Dinosaur Mysteries did you like best? 
(multiple answers accepted) 


• When asked for their favorite exhibit, the list is a little different from the tracking 
and timing and sweeps data since it includes live animals in the top favorites. 


Dig Activities (28) 

Trackway (22) 

Airscribe/drilling (10) 

Live animals (10) 

Models (5) 

Hands-on activities (5) 

Skeletons (4) 

T -Rex area (4) 

Create /Emailing dinosaur (3) 

Satellite map (TerraLink?)(2) 

Time line (2) 

Recreation of dinosaurs (2) 

Puzzle (2) 

Coloring computer (2) 

Tornado (2) 

Measuring (2) 

Videos/ movies (2) 

Big bones and skeletons you can touch (1) 


Life cycle (1) 

Maps (1) 

Globe (1) 

Trackway debate (1) 

Giant skull (1) 

Weather (1) 

Is it a dinosaur? (1) 
Asteroid debate (1) 

All of it (1) 

Extinction video (1) 

Nest (1) 

Sounds (1) 

Spin browser (1) 

Computer information (1 ) 
No specific one (1) 

Plate tectonics computer (1) 


13. Were there any exhibits that were too hard or that you couldn't get to work? 


• Only 15% of the respondents had problems with some of the interactives. 


Table 13: Problems 

Number Percent 

Yes 12 15% 

No 68 85% 


Exhibit Problem Areas 
Dinosaur song down (2) 

Some video interactives were too much for younger kids (2) 
Sensitivity of life cycle dial (A3) (1) 

Airscribe (1) 

Overall too broken up, more closed off sections (1) 

Need more instructions /docent help on videos/dig (1) 
Plates not rotating on computer interactive (1) 

Child didn't have patience for airscribe (1) 

Trackway is a bit long and confusing (1) 

Couldn't find the salamander (1) 
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14. How would you rate the Dinosaur Mysteries exhibit? Check one for each item. 

{Excellent = 5, Very Good = 4, OK = 3, Fair = 2, Poor = 1) 

• Visitors gave the exhibit very high ratings 


Mean Range 


Overall experience 

4.5 

3-5 

Hands-on activities 

4.6 

3-5 

Dig Activity Videos 

4.4 

3-5 

Scientists' Debate Videos 

4.4 

3-5 

Skeletons 

4.6 

3-5 

Full size dinosaur 

Models 

4.7 

3-5 


15. If you come back to the Maryland Science Center, will you visit this exhibit 
again? 


• Ninety-eight percent of the respondents said they would return. 



Number 

Percent 

Yes 

78 

98% 

No 

I 

1% 

No answer 

1 

1% 


Other Comments: 

Good for children. 

Trackway interpreter was very helpful. Nicely done. 
Liked all exhibit stations. 

Docents were excellent. 

Debate videos were too long. 


Conclusions 

Dinosaur Mysteries is a highly successful and popular exhibit. Visitors give it high 
ratings and spend significant amount of time in the exhibit. Visitors to this exhibit 
seem to be emphasizing quality rather than quantity. While the average visitor sees 
only 9 out of 67 exhibit components, they seem to understand the exhibit content 
and are able to explain the main messages. This may be a function of the high 
percentage of young children in the audience. Nevertheless, few visitors had any 
problems with the exhibit and almost all would return to the science museum. 
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WEDNESDAY, JUNE 6, 2007 

House of Representatives, 

Subcommittee on Research and Science Education, 

Committee on Science and Technology, 

Washington, DC. 

The Subcommittee met, pursuant to call, at 2:40 p.m., in Room 
2318 of the Rayburn House Office Building, Hon. Jerry McNerney 
[Acting Chairman of the Subcommittee] presiding. 
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Purpose 

The purpose of the hearing is to review the K-16 science, technology, engineering, 
and mathematics (STEM) education activities of federal agencies and to explore cur- 
rent efforts for the improvement of interagency coordination and evaluation of pro- 
grams. In addition, agencies will be asked to respond to the witness testimonies 
given on May 15, 2007, about the educators’ views on the STEM education programs 
at federal R&D mission agencies. The witnesses provided Subcommittee Members 
with their suggestions for how those agencies could best contribute to STEM edu- 
cation nationwide and strongly recommended closely collaborating with educators in 
the field when developing programs. 

Witnesses 

Dr. Cora Marrett, Assistant Director, Directorate for Education and Human Re- 
sources, National Science Foundation; Co-Chair, Education and Workforce Develop- 
ment Subcommittee, National Science and Technology Committee 

Dr. Joyce Winterton, Assistant Administrator, Office of Education, National Aero- 
nautics and Space Administration 

Mr. William Valdez, Director, Office of Workforce Development for Teachers and 
Scientists, Office of Science, Department of Energy 

Dr. Bruce Fuchs, Director, Office of Science Education, National Institutes of 
Health 

Overarching Questions 

• What steps have agencies taken to improve coordination with other federal 
agencies’ STEM education activities and, in particular, what is the status of 
the new coordinating committee under the National Science and Technology 
Committee (NSTC)? To what extent do agencies collaborate with educators in 
the states and school districts in developing STEM education programs? 

• The recent report of the Academic Competitiveness Council reinforces the 
need for better evaluation and performance metrics for federal STEM edu- 
cation programs. What plans do agencies have to improve evaluation of STEM 
programs? 

• The Subcommittee received testimony at a hearing on 15 May on how the 
R&D mission agencies could improve the effectiveness of their STEM edu- 
cation programs. The witnesses were skeptical of the ability of the R&D mis- 
sion agencies to develop curricular materials for formal classroom instruction 
and questioned the effectiveness of their teacher professional development 
programs to improve teacher classroom performance, while suggesting that 
the agencies’ most important role is in informal STEM education. The wit- 
nesses also strongly recommended closer collaboration by the agencies with 
educators in the field when developing STEM programs. What are agencies’ 
responses to the recommendations from these witnesses? 

• How do the agencies determine priorities for their K-16 STEM education 
portfolios? Have the agencies’ balance of programs at graduate/post doctoral, 
undergraduate, K-12, and informal education changed much over the past 
few years? Is there a likelihood of a change in that balance in the future? 
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• How do agencies disseminate information about STEM education programs? 
What organizations, both government and private, have agencies partnered 
with to reach educators in the field? 

Background 

STEM Education Funding 

In an effort to identify the contributions of federal agencies to improving STEM 
education, the Academic Competitiveness Council (ACC) was created in the Deficit 
Reduction Act of 2005 (P.L. 109-171) and charged with creating an inventory of 
STEM education programs across federal agencies, identifying the effectiveness of 
those programs, determining areas of overlap or duplication among programs, iden- 
tifying target populations served by the programs, and recommending processes to 
integrate and coordinate those programs. After a year long study, the ACC released 
a report containing an inventory of $3.12 billion in funding for Fiscal Year (FY) 
2006 for 105 STEM education programs. This inventory showed that nearly 50 per- 
cent of funding was directed toward Graduate/Post Doctoral programs ($1.4 billion) 
and another 30 percent was directed toward Undergraduate Programs ($943 mil- 
lion). K-12 programs received approximately $574 million in funding and informal 
education programs received $137 million in funding. 


ACC Report - Ending for Federal STEIfl Edl Ptogmiiia 
(FYQ6 Appropriated Fundte) 
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The agencies under the jurisdiction of the Committee on Science and Technology 
showed a similar balance of funding for STEM education programs with about two- 
thirds of funding going to post-secondary programs. The National Institutes of 
Health, whose representative has been included as a witness in this hearing due 
to agency’s large investment in STEM education, reported a total of $856 million 
in FY06 funding with 89 percent dedicated to the Kirschstein National Research 
Service Award for graduate/post doctoral fellowships ($761 million). NIH provided 
approximately $52 million for K-12 programs (six percent), $37 million for under- 
graduate programs (4.4 percent), and $5 million for informal education programs 
(0.6 percent). 
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Table 1 FY06 Appropriated Funds (in millions) for Agencies under the Jurisdiction of the 
Committee on Science and Technology 


Agency 

K-12 


Graduate/Post 

Doctoral 

Informal 

Totals 

EPA 

$0.00 

$0.00 

$11.06 

$0.00 

$11.06 

Energy 

$4.34 

$2.28 

$5.50 



NASA 

$23.00 

$0.00 

$105.40 

$34.00 

$162.40 

NIST 

$0.00 

$0.42 

$11.02 


$11.44 

NOAA 

$11.59 

3.96 

$14.65 

$7.76 


NSF 

$241.60 

$351.35 

$259.18 



Totals 

$280.53 

$358.01 

$406.81 

$113.36 

$1,158.71 


Source: Report of the Academic Competitiveness Council, May 2007 


The ACC set parameters of its inventory, limiting the programs for inclusion to 
those “primarily intended to provide support for, or to strengthen, science, tech- 
nology, engineering, or mathematics education.” The Subcommittee on Research and 
Science Education, realizing that many educational activities carried out by the fed- 
eral R&D mission agencies are contained within larger programs, worked with those 
agencies to provide a more in-depth view of those efforts. Excluding graduate edu- 
cation programs which already dominate mission agencies’ STEM funding, an addi- 
tional $256.65 million in FY06 appropriated funds for K-12, undergraduate, and in- 
formal education activities was identified for a total of $1.01 billion in K-16 funding 
at NSF and the federal R&D mission agencies. (EPA reported no FY06 funding for 
K-16 STEM education activities.) 
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K -16 STEM Ed FY06 Appropriated Funds 



Figure 2 Source: Agency submissions compiled by the Committee on Science and Technology, 
Subcommittee on Research and Science Education 

As can be seen in the table in Figure 3 K-16 STEM Ed FY06 Appropriated Funds, 
roughly an equal amount of funding is dedicated to undergraduate activities as K- 
12 and informal education activities combined. 
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Evaluation of Programs 

Another consistent criticism of federal STEM education programs is a lack of eval- 
uation methods which would show the level of effectiveness of a program. The ACC 
studied evaluation processes used by the identified STEM education programs and 
concluded in its report that “there is a general dearth of evidence of effective prac- 
tices and activities in STEM education.” The ACC recommended that funding for 
any program should not be increased until it can show effectiveness as determined 
by rigorous evaluation methods. The report points to randomized control trials or, 
when that is not feasible, well-matched comparison group studies as the optimal 
methods for determining if a program is effective. 

This sentiment was echoed by the witnesses at the Subcommittee’s hearing on 
May 15, 2007, Federal STEM Education Programs: Educators’ Perspectives , who ex- 
plained to Members that the absence of consistent performance measurements 
makes choosing among the vast array of programs difficult and time-consuming. Al- 
though all of the witnesses agreed that evaluations should be improved, two stated 
that they did not think that research methodologies such as randomized controlled 
trials would be practical or necessary for the majority of programs. Instead, they 
recommended that programs focus on developing design critiques of proposed pro- 
grams and formative evaluations of current programs to guide decisions for building 
highly effective programs. 

Coordination and Collaboration 

The federal agencies have also been criticized for their lack of coordination and 
collaboration between agencies and with state and district education agencies when 
developing programs. All of the witnesses in the May 15th hearing agreed strongly 
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with the need for more effort by the federal agencies to work with educators in the 
field. The Mathematics and Science Director of Chicago Public Schools explained 
that materials and programs developed by federal agencies that do not fit into the 
district’s curriculum and the state standards only add confusion and distract from 
the successful program they have built. The Director of Science, Technology, and 
Mathematics at Western Washington University commented, “There is a huge in- 
ventory of poorly-designed and under-evaluated mission-related curricula, posters, 
and lesson plans and associated professional development rarely used in classrooms 
and with no natural home in a coherent standards-based curriculum. The constant 
barrage of new resources adds to the noise in the system and contributes to the 
mile-wide, inch-deep problem.” 

In response to this issue, the ACC recommended that the National Science and 
Technology Committee (NSTC), which serves as the principal body for coordinating 
federal research and development, re-establish the Education and Workforce Devel- 
opment subcommittee to encourage the agencies to share knowledge and develop a 
federal strategic plan for effectively increasing STEM proficiency nationwide. The 
NSTC recently announced the subcommittee will be co-chaired by NSF, the Depart- 
ment of Education, and the National Institutes of Health. Dr. Cora Marrett, as the 
Co-Chair from NSF, has been asked to provide an update on the status of the NSTC 
subcommittee in this hearing. 

Specific Questions for the Witnesses 

Dr. Marrett 

• As Co-Chair of the NSTC Subcommittee on Education and Workforce Devel- 
opment, please describe the make up of the group, current activities, and 
planned activities. 

• What steps has your agency taken to improve its coordination with other fed- 
eral agencies’ STEM education activities? How has your agency improved its 
collaboration with states and districts in developing STEM education pro- 
grams? Please describe your agency’s commitment to establishing formal 
mechanisms to improve in these areas. 

• The ACC report reinforces the need for better evaluation and performance 
metrics for federal STEM education programs. How has your agency made 
improvements in its evaluation of programs? How has this affected your agen- 
cy’s funding for STEM education programs? 

• How does your agency determine priorities for its K-16 STEM education port- 
folio? Has your agency’s balance of programs at graduate/post doctoral, under- 
graduate, K-12, and informal education changed? Do you foresee a change in 
that balance in the future? 

• How does your agency disseminate information about its STEM education 
programs? What organizations, both government and private, have you 
partnered with to reach educators in the field? 

Dr. Joyce Winterton 

• What steps has your agency taken to improve its coordination with other fed- 
eral agencies’ STEM education activities? To what extent does your agency 
collaborate with educators in the states and school districts in developing 
STEM education programs? 

• The recent report of the Academic Competitiveness Council reinforces the 
need for better evaluation and performance metrics for federal STEM edu- 
cation programs. What plans does your agency have for improvements in its 
evaluation of its STEM programs? 

• The Subcommittee received testimony at a hearing on 15 May on how the 
R&D mission agencies could improve the effectiveness of their STEM edu- 
cation programs. (Witness statements and video of the hearing can be 
downloaded at http://www.science.house.gov/publications/hearings- mark- 
ups -details. aspx?NewsID=1814). The witnesses were skeptical of the ability 
of the agencies to develop curricular materials for formal classroom instruc- 
tion and questioned the effectiveness of their teacher professional develop- 
ment programs to improve teacher classroom performance, while suggesting 
that the agencies’ most important role is in informal STEM education. The 
witnesses also strongly recommended closer collaboration by the agencies 
with educators in the field when developing STEM programs. What is your 
response to the recommendations from these witnesses? 
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• How does your agency determine priorities for its K-16 STEM education port- 
folio? Has your agency’s balance of programs at graduate/post doctoral, under- 
graduate, K-12, and informal education changed much over the past few 
years? Do you foresee a change in that balance in the future? 

• How does your agency disseminate information about its STEM education 
programs? What organizations, both government and private, have you 
partnered with to reach educators in the field? 

Mr. William Valdez 

• What steps has your agency taken to improve its coordination with other fed- 
eral agencies’ STEM education activities? How has your agency improved its 
collaboration with states and districts in developing STEM education pro- 
grams? Please describe your agency’s commitment to improving in these 
areas. 

• The ACC report reinforces the need for better evaluation and performance 
metrics for federal STEM education programs. How has your agency made 
improvements in its evaluation of programs? How has this affected your agen- 
cy’s funding for STEM education programs? 

• In response to the testimonies given on May 15th by STEM educators, what 
do you recommend as the most effective role your agency can play in improv- 
ing STEM literacy? 

• How does your agency determine priorities for its K-16 STEM education port- 
folio? Has your agency’s balance of programs at graduate/post doctoral, under- 
graduate, K-12, and informal education changed? Do you foresee a change in 
that balance in the future? 

• How does your agency disseminate information about its STEM education 
programs? What organizations, both government and private, have you 
partnered with to reach educators in the field? 

• Please describe the process you utilized to gather information for creating a 
strategic plan for the OWDTS education programs. Include a synopsis of the 
information gathered. 

Dr. Bruce Fuchs 

• What steps has your agency taken to improve its coordination with other fed- 
eral agencies’ STEM education activities? To what extent does your agency 
collaborate with educators in the states and school districts in developing 
STEM education programs? 

• The recent report of the Academic Competitiveness Council reinforces the 
need for better evaluation and performance metrics for federal STEM edu- 
cation programs. What plans does your agency have for improvements in its 
evaluation of its STEM programs? 

• The Subcommittee received testimony at a hearing on 15 May on how the 
R&D mission agencies could improve the effectiveness of their STEM edu- 
cation programs. (Witness statements and video of the hearing can be 
downloaded at http://www.science.house.gov/publications/hearings- mark- 
ups -details.aspx?NewsID=1814). The witnesses were skeptical of the ability 
of the agencies to develop curricular materials for formal classroom instruc- 
tion and questioned the effectiveness of their teacher professional develop- 
ment programs to improve teacher classroom performance, while suggesting 
that the agencies’ most important role is in informal STEM education. The 
witnesses also strongly recommended closer collaboration by the agencies 
with educators in the field when developing STEM programs. What is your 
response to the recommendations from these witnesses? 

• How does your agency determine priorities for its K-16 STEM education port- 
folio? Has your agency’s balance of programs at graduate/post doctoral, under- 
graduate, K-12, and informal education changed much over the past few 
years? Do you foresee a change in that balance in the future? 

• How does your agency disseminate information about its STEM education 
programs? What organizations, both government and private, have you 
partnered with to reach educators in the field? 



192 


Mr. McNerney. Welcome to today’s hearing, entitled “Federal 
STEM Education Programs.” I personally want to thank everyone 
for coming here today. It is a nice audience for the proceedings of 
the Subcommittee on Research and Science Education. 

This hearing is part of an ongoing assessment that the Com- 
mittee is undertaking to determine the role of the Federal Govern- 
ment in science, technology, engineering, and math education. We 
will be hearing today from four federal agencies about their 
progress in improving STEM programs. As a mathematician and 
someone who believes strongly in the need to expand educational 
opportunities for the next generation, today’s hearing holds par- 
ticular importance for me. 

In the past few months, this committee has done a commendable 
job in drawing attention to the difficulties that our country will 
face if we are unable to increase the number of future scientists 
and researchers. From the difficult but not insurmountable chal- 
lenges laid out in the National Academies’ “Rising Above the Gath- 
ering Storm” report, to witness testimony from businesses and uni- 
versities, we have heard time and again that we need more STEM 
education graduates and teachers if we are to compete successfully 
in the global economy. 

This committee, under the leadership of Chairman Gordon, has 
succeeded in passing legislation that will pay significant future 
dividends, and will be a great legacy for this Congress. The issues 
we are discussing today are no less important. By hearing from 
educators on the front lines of the educational system, as well as 
the federal agencies that must implement STEM programs, we are 
laying out the groundwork for how to maximize the benefits of ex- 
isting programs, which will invariably lead to future initiatives. 

Last month, the Subcommittee held a hearing with local STEM 
educators to learn their perspectives on federal programs. During 
witness questioning, I made a point to highlight what I believe to 
be one of the most important aspects of any discussion on STEM 
education, and that is how we can reach more students, and make 
sure that the United States is not only keeping up with the rest 
of the world, but is outpacing other countries. It is hard work to 
earn a degree in math or physics, as our Ranking Member might 
testify, and we need to make sure that our federal policies first and 
foremost benefit our students. 

Our previous witnesses shared with us their recommendations 
for STEM education improvements. That included working with 
states and districts to align federal programs to local standards. 
The witnesses also expressed frustration that there are currently 
no consistent guidelines for evaluating STEM programs across gov- 
ernment agencies, and we have asked today’s witnesses to respond 
to the comments and recommendations from the previous panel. 

In addition, the week before last month’s hearing, the Academic 
Competitiveness Council released a comprehensive report on fed- 
eral STEM education programs. Not surprisingly, the ACC assess- 
ment was the same as that of our witnesses. Collaboration between 
agencies and with State and local governments needs improvement. 
Programs should be designed around best practices, and a wide va- 
riety of educational measurements should be simplified into com- 
mon ones. Each of the agencies invited to testify this afternoon 
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were deeply involved in the ACC’s process, and we have asked each 
of them to communicate today what changes they have made over 
the past year in order to achieve these goals. 

I am hopeful that today’s witnesses will shed some light on how 
we have been spending scarce federal dollars on STEM education, 
and how we can improve our current practices so that we are bene- 
fiting students from kindergarten all the way up. I look forward to 
the discussion with our witnesses and how their agencies are plan- 
ning, coordinating, and evaluating their efforts to improve STEM 
education. 

The Chair now recognizes Dr. Ehlers for an opening statement. 
[The prepared statement of Vice Chairman McNerney follows:] 

Prepared Statement of Vice Chairman Jerry McNerney 

Good afternoon, and thanks to everyone for attending today’s proceedings of the 
Subcommittee on Research and Science Education. This hearing is part of an ongo- 
ing assessment that the Committee is undertaking to determine the role of the Fed- 
eral Government in science, technology, engineering, and math education. We’ll be 
hearing today from four federal agencies about their progress improving STEM pro- 
grams. As a Mathematician, and someone who believes strongly in the need to ex- 
pand educational opportunities for the next generation, today’s hearing holds par- 
ticular importance for me. 

In the past few months, this committee has done a commendable job in drawing 
attention to the difficulties our country will face if we are unable to increase the 
number of future researchers and scientists. From the difficult — but not insur- 
mountable — challenges laid out in the National Academies’ Rising Above the Gath- 
ering Storm report to witness testimony from businesses and universities, we’ve 
heard time and again that we need more STEM educated graduates and teachers 
if we can compete in a global economy. 

This committee — under the leadership of Chairman Gordon — has succeeded in 
passing legislation that will pay significant future dividends, and will be a great leg- 
acy for Congress. The issues we’re discussing today are no less important. By hear- 
ing from educators on the front lines of the education system as well as the federal 
agencies who must implement STEM programs, we’re laying the groundwork for 
how to maximize the benefit of existing programs, which will invariably improve 
future initiatives. 

Last month, the Subcommittee held a hearing with local STEM educators to learn 
their perspectives on federal programs. During witness testimony and in ques- 
tioning, I made it a point to highlight what I believe to be one of the most important 
aspects of any discussion on STEM education and that is how we can reach more 
students and make sure that the United States is not only keeping up with the rest 
of the world, but out-pacing other countries. It is hard work to earn a degree in 
math or physics, and we need to make sure that our federal policies — first and fore- 
most — benefit students. 

Our previous witnesses shared with us their recommendations for STEM edu- 
cation improvements that included working with states and districts to align federal 
programs to local standards. The witnesses also expressed frustration that there are 
currently no consistent guidelines for evaluating STEM programs across govern- 
ment agencies, and we have asked today’s witnesses to respond to the comments 
and recommendations from the previous panel. 

The week before last month’s hearing, the Academic Competitiveness Council re- 
leased a comprehensive report on federal STEM education programs. Not surpris- 
ingly, the ACC assessment was the same as that of our witnesses — collaboration be- 
tween agencies and with State and local governments needs improvement, programs 
should be designed around best practices, and a wide array of evaluation measure- 
ments should be simplified into common ones. Each of the agencies invited to testify 
this afternoon were deeply involved in the ACC’s process and we have asked each 
of them to communicate today what changes they have made over the past year in 
order to achieve these goals. 

I’m hopeful that today’s witnesses will shed some light on how we’ve been spend- 
ing scarce federal dollars on STEM education, and how we can improve on current 
practices so that we’re benefiting students from kindergarten and up. I look forward 
to the discussion with our witnesses on how their agencies are planning, coordi- 
nating, and evaluating their efforts to improve STEM education. 
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Mr. Ehlers. Thank you, Mr. Chairman. It is a pleasure to be 
here to participate in yet another event centered on STEM edu- 
cation. You and I, I think, have both spent a good part of our lives 
dedicated to education, particularly science, technology, engineer- 
ing, and math education. 

I do have to clarify that what the S, T, M, and E stand for, even 
though everyone in here should be familiar with it, but I have to 
do that for anyone who isn’t familiar, because tomorrow, we are 
going to have debate, discussion, and votes on two stem cell bills, 
and so, all the papers are going to talk about for the next few days 
is votes on stem cell bills. We are not talking about a stem cell bill 
here today. In fact, I think this is, for the future of our nation, 
probably considerably more interesting. 

STEM education is a priority for this nation. It has to be if our 
nation is to survive, to continue to improve in the welfare of its 
own citizens, and frankly, also in the welfare of the world. Thanks 
to constantly increasing understanding of the importance of STEM 
education to our national competitiveness, I no longer have to go 
into a lot of details on this, but even with this improving aware- 
ness of STEM education, there is still more that the Federal Gov- 
ernment can do to improve K-12 STEM education in the United 
States, and we clearly need improvement, because we are still 
ranked near the bottom of the developed countries in STEM edu- 
cation. 

The Academic Competitiveness Council was created by Congress 
to catalog and coordinate the STEM education projects and pro- 
grams currently supported by the Federal Government. I commend 
the agencies that participated in this endeavor. For some time, I 
have been very concerned about the fact that the Federal Govern- 
ment and its agencies and also some private entrepreneurs in in- 
dustry are developing these fantastically good programs, but there 
is no overall correlation of how they should be used in the class- 
room, how they fit in with the curriculum, and I think we should 
be working on that. 

I commend the agencies that have participated in this endeavor. 
The charge to the Academic Competitiveness Council was a chal- 
lenging one, and the report reflects the breadth and depth of pro- 
grams that exist at our federal agencies. From the start, I harbored 
a general concern that the ACC might overzealously seek out seem- 
ingly duplicative programs, and inadvertently encourage their de- 
mise, which happened a few years ago with the Math Science Part- 
nerships. Someone in the basement of the White House noticed 
that we had two Math Science Partnerships, one in the National 
Science Foundation, one in the Department of Education, and said, 
“A ha, duplication, we must get rid of one.” Fortunately, this has 
not happened in your case on these issues. 

I think the ACC report sheds light on the diversity and unique- 
ness of the programs that are developed, and sends a clear message 
that Congress must authorize adequate evaluation capacity for fed- 
eral STEM education programs. It is crucial that we evaluate these 
programs with the most appropriate and rigorous techniques avail- 
able. Overall, the ACC report provides a useful foundation for fu- 
ture coordination and collaboration, so that federal agencies can 
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work together to leverage STEM resources and communicate suc- 
cesses as well as failures. 

I am pleased that the recently reestablished National Science 
and Technology Council Subcommittee on Education and Workforce 
Development will follow through on actions recommended by the 
ACC. 

I look forward to hearing from our witnesses today about how 
they are moving towards increasing collaboration, as well as be- 
coming more educated about the STEM education programs at 
their respective agencies. 

With that, I yield back the balance of my time. 

[The prepared statement of Mr. Ehlers follows:] 

Prepared Statement of Representative Vernon J. Ehlers 

STEM education is a priority for this nation. Thanks to a constantly increasing 
understanding of the importance of STEM to our national competitiveness, I no 
longer must define what the “S,” “T,” “E” and “M” stand for; today, my colleagues 
are familiar with the acronym. Even with this improving awareness of STEM, there 
is still more that the Federal Government can do to improve K-16 STEM education 
in the U.S. 

The Academic Competitiveness Council (ACC) was created by Congress to catalog 
and coordinate the STEM education projects and programs currently supported by 
the federal government. I commend the agencies that participated in this endeavor. 
The charge to the Council was a challenging one, and the report reflects the breadth 
and depth of programs that exist at our federal agencies. From the start, I harbored 
a general concern that the ACC might overzealously seek out seemingly duplicative 
programs and inadvertently encourage their demise. Instead, I think the ACC report 
sheds light on the diversity and uniqueness of programs, and sends a clear message 
that Congress must authorize adequate evaluation capacity for federal STEM edu- 
cation programs. It is crucial that we evaluate these programs with the most appro- 
priate and rigorous techniques available. 

Overall, the ACC report provides a useful foundation for future coordination and 
collaboration, so that federal agencies can work together to leverage STEM re- 
sources and communicate successes as well as failures. I am pleased that the re- 
cently re-established National Science and Technology Council (NSTC) will follow 
through on actions recommended by the ACC. 

I look forward to hearing from our witnesses today about how they are moving 
toward increased collaboration, as well as becoming more educated about the STEM 
education programs at their respective agencies. 

Mr. McNerney. Thank you. The Chair thanks the gentleman 
from Michigan. If there are any Members, I don’t see any here, but 
if there are any Members who wish to submit additional opening 
statements, your statements will be added to the record at this 
point. 

[The prepared statement of Ms. Johnson follows:] 

Prepared Statement of Representative Eddie Bernice Johnson 

Thank you, Mr. Chairman. 

Today’s hearing will be valuable to determine how to help the federal science 
agencies become more efficient in working together in their efforts toward STEM 
education. 

As you know, the Academic Competitiveness Council recently studied evaluation 
processes used by the various federal STEM education programs and concluded in 
its report that “there is a general dearth of evidence of effective practices and activi- 
ties in STEM education.” 

The ACC recommended that funding for any program should not be increased 
until it can show effectiveness as determined by rigorous evaluation methods. 

Witnesses at the May 15, 2007, Subcommittee hearing on “Federal STEM Edu- 
cation Programs: Educators’ Perspectives” concurred with this sentiment. 

The absence of consistent performance measurements makes choosing among the 
vast array of programs difficult and time-consuming. 



196 


Mr. Chairman, it can safely be said that all Members of this subcommittee care 
deeply about our scientific enterprise and are committed to supporting and stream- 
lining it to most responsibly use taxpayer dollars. 

We hope to determine, from witness feedback, how to work toward that goal. 

Again, welcome to today’s witnesses. Thank you, Mr. Chairman. I yield back. 

[The prepared statement of Mr. Carnahan follows:] 

Prepared Statement of Representative Russ Carnahan 

Mr. Chairman, thank you for hosting this hearing to examine the participation 
of federal agencies in STEM education and investigate approaches to improving co- 
ordination and evaluation of their programs. 

As we have all mentioned time and again, the Rising Above the Gathering Storm 
report provided us with both the knowledge that our nation’s standing as the global 
leader in the STEM field is at risk as well as solid tools for policy-makers to coun- 
teract this worrisome trend. Chairman Gordon, you have been a tremendous advo- 
cate for improving STEM education in this nation. I am proud to be a Member of 
the Committee on Science and Technology — under your leadership we have success- 
fully moved four major innovation initiatives through the House just in these past 
couple months. 

I am pleased that today’s hearing again focuses on the important task of ensuring 
that our STEM programs are working to the best of their abilities. 

I am eager to hear our witnesses’ assessments of these agencies’ contributions to 
STEM education programs so that we can reflect on the successes and inefficiencies 
of the programs and seek to make modifications for improvement. Your first-hand 
experiences are vital to maximizing the resources we are offering our nation’s aspir- 
ing students. 

To all the witnesses — thank you for taking time out of your busy schedules to ap- 
pear before us today. I look forward to hearing your testimony. 

Mr. McNerney. At this time, I would like to introduce our dis- 
tinguished witnesses. First, we have Dr. Cora Marrett, is that 
being pronounced correctly? She is the Assistant Director for the 
National Science Foundation Directorate for Education and Human 
Resources. Dr. Marrett is also the Co-Chair of the Education and 
Workforce Development Subcommittee under the NSTC. Welcome 
aboard, Dr. Marrett, and she is going to be chairing the new sub- 
committee, so it is a big burden on your shoulders, and I am look- 
ing forward to your words. 

Our second panelist is Dr. Joyce Winterton. She is the Assistant 
Administrator of NASA’s Office of Education. Welcome. 

Third, we have Mr. Bill Valdez, Director of Office for Workforce 
Development for Teachers and Scientists at the Department of En- 
ergy. Welcome, this afternoon. 

And finally, we have Dr. Bruce Fuchs, and he is the Director of 
the Office of Science Education at the National Institutes of 
Health. 

Welcome all, and as our witnesses know, spoken testimony is 
limited to five minutes each, after which each Member of the Com- 
mittee will have five minutes to ask questions. We will try and 
limit you to five minutes, but I understand if you have a few extra 
minutes to run over, but we do want to keep things in line. 

So, we will start with Dr. Marrett at this point. Would you begin 
your testimony? 

STATEMENT OF DR. CORA B. MARRETT, ASSISTANT DIRECTOR, 

EDUCATION AND HUMAN RESOURCES DIRECTORATE, NA- 
TIONAL SCIENCE FOUNDATION 

Dr. Marrett. Thank you very much, Chairman McNerney, and 
Ranking Member Ehlers. I do appreciate the opportunity today, 
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and want to express my gratitude to the entire Subcommittee, for 
your longstanding support for excellence in science, technology, en- 
gineering, mathematics education, or STEM. 

The National Science Foundation appreciates the interest ex- 
pressed by this subcommittee and others in coordination and eval- 
uation of STEM-centered activities. In fact, the National Founda- 
tion, or NSF, takes pride in the actions we have undertaken over 
the years to enhance excellence across all levels of education and 
all fields of science and engineering. 

We owe our successes to the interactions we have had with com- 
munities of researchers, educators, diverse organizations, and in- 
deed, other agencies. Those interactions have shaped significantly 
the content of our efforts, our evaluation of them, and our dissemi- 
nation strategies. We are aware, however, that we must revisit con- 
tinually the approaches and connections we cultivate. 

This said, these are some of the ways in which we are looking 
at what is the current situation, and how we respond to the chal- 
lenges that now exist. The reconstitution of the Subcommittee on 
Education and Workforce Development of the National Science and 
Technology Council, to which you have just referred, that recon- 
stitution should help us in NSF strengthen our ties to other federal 
agencies. I should note we already have a number of ties. One of 
our most recent is a partnership through a memorandum of under- 
standing with NASA, and I am delighted that Dr. Winterton and 
I had an opportunity quite recently to host a conference in which 
we had participation from our other colleagues here at the table 
and other agencies. 

The subcommittee, the reconstituted subcommittee of the NSTC, 
the membership for it will come from the agencies represented on 
NSTC’s larger Committee on Science. The representatives are to 
possess substantive knowledge of their STEM education portfolio 
within the agency, and these representatives are to have experi- 
ence with evaluation research and, possibly, with the development 
and application of performance measures. So, in looking at what we 
need for that committee, we realize it is knowledge both of what 
agencies are doing, but familiarity to work on these very important 
issues of evaluation research performance measures. 

The subcommittee that I am referring to will address a broad 
range of issues related to STEM education. To use a phrase that 
is often heard these days, this will be attending from K to gray, 
because the education portfolio does indeed cover all phases of edu- 
cation and workforce activities. The subcommittee will provide a 
forum for exchanges of information and expertise regarding re- 
search and evaluation. On the one side, then, we have the develop- 
ment, the reconstitution of the Subcommittee on Education and 
Workforce Development. 

Another important development, of course, is the report you have 
referred to from the Academic Competitiveness Council, or the 
ACC. As we look at the report, it certainly serves to enhance our 
attention to evaluation. Now, within the National Science Founda- 
tion, there has been a requirement that there is evaluation associ- 
ated with every program within the Directorate of Education and 
Human Resources. In fact, Congress had a lot to do with the direc- 
tive that set this as expected. 
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We have responded and indeed now have a requirement that all 
such programs within this part of the organization do have to be 
subject to evaluation. But we have other work to do, and we see 
ourselves as attending increasingly to such matters as clarifying 
the objectives of particular programs. Evaluation must depend on 
how clear is the intent of any activity. We see ourselves as attend- 
ing far more perhaps to the definition of concepts, and ensuring 
that those definitions are shared, especially with the other agencies 
that must be involved. These become fundamental issues to ensure 
that we will be able to conduct the rigorous evaluations that we 
agree must be essential for determining how resources are being 
used. In collaboration with others, we will strive to enhance the ca- 
pacity for and knowledge base of such evaluation. There is a need 
for expanding the community, the experts who can, in fact, bring 
to the matter of evaluation and research the strong conceptual the- 
oretical work that is essential. 

There are, in addition to these matters of evaluation, extensive 
possibilities for expanded opportunities to improve STEM edu- 
cation, opportunities for engaging with others on research, for ex- 
ample. The hearing that you had with STEM educators, the inquir- 
ies we have received from foundations, lots of private foundations, 
industrial groups, the responsiveness we have received from school 
districts and higher education associations, all of these develop- 
ments prompt our heightened attention and commitment to collabo- 
ration, collaboration in the cause of excellence. 

In closing, then, we at the National Science Foundation will not 
rest on our past achievements. Rather, we will continue to foster 
and tap the creativity this Nation needs for the success of our citi- 
zenry in the years ahead. I am willing, of course, to respond to 
questions. This could be nothing more than a quick snapshot of 
what we have in mind, what we have done, and where we intend 
to go at the National Science Foundation. 

Thank you. 

[The prepared statement of Dr. Marrett follows:] 

Prepared Statement of Cora B. Marrett 

Chairman Baird, Ranking Member Ehlers, and Members of the Subcommittee. 
Thank you very much for inviting me to testify before you today on science, tech- 
nology, engineering, and mathematics (STEM) education. 

This subcommittee’s commitment to excellence in STEM education at the National 
Science Foundation (NSF) is well known, and we are extremely appreciative of your 
long-standing support. 

As you are well aware, the NSF provides leadership at the federal level to ad- 
vance learning and discovery in all disciplines of science and engineering and to fos- 
ter connections among the disciplines. The Director of NSF, Dr. Arden Bement, has 
presented the case eloquently: “Our job is to keep science and engineering vision- 
aries focused on the furthest frontier, to recognize and nurture emerging fields, to 
prepare the next generation of scientific talent, and to ensure that all Americans 
gain an understanding of what science and technology have to offer.” 

The questions for today’s hearing center on the coordination of STEM-related pro- 
grams, the evaluation of those programs, and the dissemination of information 
about effective strategies. These long have been central concerns for NSF, as is evi- 
dent in activities we have undertaken over the years. But we are cognizant of 
changes looming on the horizon that will require heightened attention to coordina- 
tion, research and evaluation as well as dissemination. 

On Coordination and Collaboration. 

The National Science Foundation works in partnership with the research and edu- 
cation community to promote excellence. Hence, for us effectiveness is indicated in 
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no small part by the connections we establish and maintain with researchers and 
educators as well as with agencies and organizations that share our commitment 
to excellence in STEM education. We seek opportunities to foster exchanges on mat- 
ters critical to such excellence. An example: the conference held recently on state 
standards for mathematics. What gave rise to the conference were the development 
by states of different standards, the efforts of several national organizations to align 
those standards, and the interest of state supervisors of mathematics in exchanging 
ideas and experiences. The conference, held in February 2007, featured presen- 
tations on recommendations regarding standards and engaged “users” of stand- 
ards — State and district curriculum specialists, textbook and assessment publishers, 
K-12 district and teacher leaders, and representatives from higher education and 
business. The National Science Foundation served as a co-sponsor of the conference, 
along with Achieve, Inc., the American Statistical Association, the College Board, 
the Mathematical Association of America, and the National Council of Teachers of 
Mathematics. The idea for the conference emerged from an NSF-sponsored entity: 
the Center for the Study of the Mathematics Curriculum at Michigan State Univer- 
sity. That Center organized the conference, in concert with the State Supervisors 
of Mathematics. The case illustrates that NSF takes a broad approach to the chal- 
lenges associated with coordination and collaboration. 

Our approach to coordination and collaboration extends beyond the formal edu- 
cation sector to include important activities in promoting understanding of science 
in the wider public. Towards that end, NSF organized in March 2007 a workshop 
on informal science activities conducted through science centers, museums, commu- 
nity projects and the media. The workshop brought designers of informal science ini- 
tiatives together with program evaluators, to generate guides for the evaluation of 
such initiatives. The workshop included representatives from other federal agencies. 
Again, the action reinforces the theme that NSF supports coordination through out- 
reach — to various communities and agencies — on matters relevant to STEM edu- 
cational policies and practices. 

The informal science workshop demonstrates, too, that NSF both endorses and 
seeks to provide leadership on program evaluation. The evaluation efforts are tai- 
lored to the goals and state of development for any given program. Moreover, NSF 
invests in research and evaluation, not just to assess outcomes, but also to build 
knowledge about and a community prepared to advance STEM research and evalua- 
tion. 

A distinctive feature of the NSF STEM education portfolio is its breadth. Not only 
does it incorporate program development as well as research, and the informal as 
well as formal sectors; it addresses the pre-college realm, undergraduate and grad- 
uate education, post-doctoral experiences, and the STEM workforce of the Nation. 
This breadth has profound implications for the collaborations NSF undertakes, the 
evaluations it supports, and the dissemination strategies it pursues. 

Subcommittee Questions 

Having provided a general context for the questions central to this hearing, let 
me now turn more specifically to those questions. 

1. As Co-Chair of the NSTC Subcommittee on Education and Workforce Develop- 
ment, please describe the make-up of the group, current activities, and planned 
activities. 

In response to the Academic Competitiveness Council (ACC) report, the Sub- 
committee is being re-constituted through representation from the agencies that 
comprise the Committee on Science of the National Science and Technology Council 
(NSTC). The representatives are to possess (1) substantive knowledge of STEM edu- 
cation programs within the given agency’s portfolio, and (2) experience with evalua- 
tion research and/or the development and application of performance measures. 
These requirements will enable the Subcommittee to meets its initial goal to coordi- 
nate and facilitate implementation of the ACC recommendations. The Subcommittee 
is also expected to address a range of issues related to STEM education at all levels. 

2. What steps has your agency taken to improve its coordination with other federal 
agencies’ STEM education activities ? How has your agency improved its collabora- 
tion with states and districts in developing STEM education programs ? Please de- 
scribe your agency’s commitment to establishing formal mechanisms to improve 
in these areas. 

Past coordination activities include formal memoranda of understanding with the 
Department of Education (ED) and the National Institutes of Health (NIH) in 1992 
and with the Department of Energy in 1995. Through the Interagency Educational 
Research Initiative, launched in 1999, NSF, NIH, and the U.S. Department of Edu- 
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cation sponsored a program of research designed to develop and/or investigate the 
effectiveness of educational interventions in classrooms across the United States. 

Earlier this year, NSF signed a memorandum of understanding on STEM edu- 
cation cooperation with National Aeronautics and Space Administration (NASA). 
The goal of this partnership is to support the development of a creative and diverse 
engineering workforce that comprehends the technical and social impacts of tech- 
nology applications and needs in a rapidly changing environment. Interactions with 
NASA precede the memorandum, however, and include our joint participation on a 
task force “to examine the feasibility and benefits of using a portion of the Inter- 
national Space Station payload resources and accommodations for education.” 

Among the ways in which NSF cooperates with the Department of Education, 
these especially warrant notice. A memorandum of understanding enabled the NSF 
and ED to fund jointly two of the large projects in our Math and Science Partner- 
ship (MSP) programs. Moreover, almost two-thirds of the sites in the NSF portfolio 
have some involvement as well with the state MSP projects that ED supports. 

In 2005 the U.S. Department of Education and the Education and Human Re- 
sources (EHR) Directorate of the National Science Foundation began collaborating 
on a Mathematics Education Toolkit. The Toolkit provides resources for state and 
district leaders on how to improve mathematics teaching and learning for Title I 
students. The Toolkit represents a response to concerns that states and districts 
have expressed. The workshop on standards, cited earlier, provides another example 
of the NSF connections beyond the federal level. 

The coordination challenges in the years ahead will extend beyond those found 
among federal agencies. Increasingly, foundations and corporations are investing in 
STEM education and the workforce. The National Science Foundation has a leader- 
ship role within the ACC and is committed to establishing whatever connections and 
mechanisms offer heightened possibilities for innovation in STEM education within 
the United States. 

3. The ACC report reinforces the need for better evaluation and performance metrics 
for federal STEM education programs. How has your agency made improvements 
in its evaluation of programs ? How has this affected your agency’s funding for 
STEM education programs ? 

The emphasis in NSF on program evaluation precedes the ACC report. A Congres- 
sional mandate in 1992 set in motion a systematic plan for assessment of programs 
within the EHR portfolio. The approach has evolved quite significantly over time, 
from one focused largely on the monitoring of developments to evaluations of im- 
pacts. The evolution has reinforced the importance of enhanced capacity for evalua- 
tion of STEM programs and accounts, then, for investments NSF has made in in- 
creasing expertise on evaluation. 

EHR education programs require project and program evaluations, and there is 
now greater emphasis on collecting evaluation information at the start of a program. 
The evaluation of a program’s value, worth, and impacts is based on a multiplicity 
of assessment and review studies. NSF evaluation efforts range from periodic meas- 
ures of project activities to in-depth analyses of a program’s success. Quantitative 
and qualitative data are obtained to measure a program’s success in achieving its 
goals. 

Our current approach encompasses a multiple method evaluation framework that 
combines theory and research to better understand and assess the R&D educational 
investment. This methodological pluralism enables programmatic decision-making to 
be based on the preponderance of the evidence from external studies. Through the 
NSTC, we will work to improve evaluation for STEM education initiatives across the 
Federal Government, including at NSF, to ensure that the most rigorous methods 
appropriate are used to assess federal programs. 

4. How does your agency determine priorities for its K-16 STEM education portfolio ? 
Has your agency’s balance of programs at graduate / post doctoral, undergraduate, 
K-12, and informal education changed? Do you foresee a change in that balance 
in the future? 

Issues for the K-16 STEM education portfolio emerge from various sources. The 
staff within NSF consists of specialists on STEM education within given disciplines, 
researchers with on-going connections to resources and knowledge, and experts on 
trends in STEM education in the United States and elsewhere. The panels that re- 
view proposals, the Committees of Visitors for our programs, and the Advisory Com- 
mittees for each directorate keep us abreast of developments and interests. In recent 
years, reports on STEM education have yielded many recommendations, as have the 
priorities established in both the Executive and Legislative branches. In deter- 
mining priorities for NSF funding, consideration is given to the capacity of external 
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communities to pursue given lines of inquiry, the activities underway through other 
agencies and organizations, and the appropriateness of the topics for the NSF port- 
folio. 

The Foundation strives to address a broad portfolio for STEM education, but does 
not have a formula for investments at each educational level. The substance of those 
investments does not remain static, however, for it must reflect changes over time 
in knowledge, national needs, and capacities within our communities and NSF. 

5. How does your agency disseminate information about its STEM education pro- 
grams ? What organizations, both government and private, have your partnered 
with to reach educators in the field ? 

The National Science Foundation disseminates information about its programs 
and the results of its investments through various channels. There are websites for 
particular programs. These include IGERT.ORG, a website produced by the Integra- 
tive Graduate Education Research and Traineeship (IGERT) program that seeks to 
attract to STEM research groups now under-represented in science and engineering. 
Communication and collaboration among MSP partners is promoted through 
MSPNet. Similarly, the Center for Learning in Out of School Environments 
(UPCLOSE) at the University of Pittsburgh serves to link researchers and educators 
who want to enhance teaching and learning in informal environments. 

Publications from the National Academy of Sciences serve to share widely the re- 
sults from NSF-investments. Among these: the path-breaking volumes, Adding It 
Up: Helping Children Learn Mathematics, and Taking Science to School. 

Our outreach efforts are extensive. What we intend to undertake in the near fu- 
ture is an assessment of the effectiveness of our strategies in reaching under-served 
communities — of educators, researchers, and institutions. Such an assessment, to be 
pursued in connection with our panels, advisory communities, and public and pri- 
vate partners, may result in modifications to our outreach efforts. 

We in NSF will not rest on past achievements or accolades. Rather, we will con- 
tinue to strive to foster and tap the creativity this nation needs for the success of 
our citizenry in the years ahead. 

Biography for Cora B. Marrett 

Dr. Cora B. Marrett is the Assistant Director of the Directorate for Education and 
Human Resources (EHR) at the National Science Foundation (NSF). She leads the 
NSF’s mission to achieve excellence in U.S. science, technology, engineering and 
mathematics (STEM) education with oversight of a budget of approximately $800 
million and a staff of 150. EHR is the principal source of federal support for 
strengthening STEM education through education research and development (R&D). 

Prior to her appointment at the NSF, Dr. Marrett served as the Senior Vice Presi- 
dent for Academic Affairs in the University of Wisconsin System. Her NSF position 
is in conjunction with the UW-Madison Department of Sociology, where she re- 
mains a tenured faculty member. 

Earlier, she held the post of Senior Vice Chancellor for Academic Affairs and Pro- 
vost at the University of Massachusetts-Amherst. 

Her current position represents a return to NSF. From 1992-1996, she served at 
NSF as the first Assistant Director of the Directorate for Social, Behavioral and Eco- 
nomic Sciences. She received the NSF’s Distinguished Service Award for her leader- 
ship in developing new research programs and articulating the scientific projects of 
the directorate. 

In addition to her faculty appointment at the University of Wisconsin-Madison, 
she has been a faculty member at the University of North Carolina and Western 
Michigan University. 

Dr. Marrett holds a B.A. degree from Virginia Union University, and M.A. (1965) 
and Ph.D. (1968) degrees from UW-Madison. She has an honorary doctorate from 
Wake Forest University (1996). She is a Fellow of the American Association for the 
Advancement of Science, the American Academy of Arts and Sciences, and Sigma 
Xi, the Science Research Society. 

In 2005, Dr. Marrett received the Erich Bloch Distinguished Service Award from 
the Quality Education for Minorities (QEM) Network, given annually to an indi- 
vidual who has made singular contributions to the advancement of science and to 
the participation of groups under-represented in science, technology, engineering 
and mathematics. She is widely published in the field of sociology, and has held a 
number of public and professional service positions. 

Mr. McNerney. Thank you, Dr. Marrett. Now, we will recognize 
Dr. Winterton. 
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STATEMENT OF DR. JOYCE L. WINTERTON, ASSISTANT ADMIN- 
ISTRATOR, OFFICE OF EDUCATION, NATIONAL AERO- 
NAUTICS AND SPACE ADMINISTRATION 

Dr. WlNTERTON. Thank you, Chairman McNerney and Congress- 
man Ehlers. I appreciate the opportunity to testify today, and have 
a discussion that we know is so important around STEM education. 

NASA certainly recognizes the role that education has in pro- 
viding that next generation of scientists, engineers, and people that 
are experts in technology, that will really advance the Nation’s eco- 
nomic well-being. The United States does have a tremendous need 
to sustain our competitive international collaborations, and we 
want to keep United States’ preeminence in that area. NASA cer- 
tainly serves as a contributor towards that goal. 

As was mentioned by Dr. Marrett, we have a growing number of 
agency collaborations: our memorandum of understanding with the 
National Science Foundation; also, we recently signed a memo- 
randum of understanding with the Federal Aviation Administra- 
tion. So, it was very rewarding, as we signed that, to see a group 
of middle school students experiencing our Smart Skies Initiative, 
that is a simulated online resource, where students get a feeling of 
what it is like to actually land a sequence of planes safely, using 
mathematics as part of that learning, and it was very interesting 
to see their motivation of seeing the real world context, and how 
you apply your math in a type of job that is in demand. 

I do serve on the interagency taskforce to revitalize the aerospace 
workforce that has been mandated by Congress, and through that, 
we are collaborating with the U.S. Department of Labor, the NSF, 
NIH, Department of Energy, Department of Defense, and looking 
forward to a substantial look at where we are currently since the 
Gathering Storm report, and what needs to be done to reenergize, 
and make sure we are on target with those. 

We certainly look at how we can work with other agencies. For 
instance, recently, at Johnson Space Center, we conducted a teach- 
er-to-teacher training that is part of the U.S. Department of Edu- 
cation’s initiative to provide professional development, and I think 
we had over 350 teachers who signed up and participated in that 
effort in Texas. 

As are other agencies, we have a renewed effort to re-look at 
what are vigorous metrics to evaluate the effectiveness of our pro- 
gram, the efficiency, but also, the long-term impact. How do we 
know that we are really investing our dollars in the right place? 
So, we actually have a specific schedule of how we will be looking 
at each of our major programs, and looking at the impact of those, 
as well. 

So, we are looking forward to our National Academies study on 
our pre-college programs that will be available in November, and 
again, that will be another opportunity for us to see how we are 
doing with those programs, how we can improve, and how we are 
meeting our customers’ needs. 

The role of our agency certainly includes professional develop- 
ment. Being a former high school teacher, that is close to my heart, 
and a teacher educator. And it is very important for NASA to work 
closely with educators as we develop resources in a formal K-12 
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setting, or as we work with our informal partners in museums and 
community-based groups like Boy Scouts, Girl Scouts, and 4-H. 

Some of the things that we are doing, for instance, with our edu- 
cator astronaut, who is a mission specialist, and will be part of our 
STS-118 launch in August, we have worked with the International 
Technology Education Association and the National Science Teach- 
ers Education, to develop resources so students can actually de- 
velop a growth chamber. They can do the engineering. Obviously, 
that is a skill we may need some day if we plan to go back to the 
Moon and beyond. So, students can use their science background, 
their engineering, at an elementary, middle school, and high school 
level to develop growth chambers, and then, we actually have basil 
seeds that will be flying on the Shuttle, that they can test, and see 
if their growth chamber will actually work. So, that is an example 
of the collaboration, making sure that we are connecting to the ex- 
isting curriculum, bringing our resources that fit the standards, 
and what teachers can afford to do, both cost-effective and time- 
wise, because we know that is very important in the curriculum 
today. 

Our NASA Explorer Schools is an example of how, over three 
years, we work with a team within a school. That is a competitive 
process. Those educators help us determine what are the needs in 
that particular school, how we can meet their needs, but also, how 
we sustain that afterwards. In fact, at the National Science Teach- 
ers Association Conference, I had a teacher from Kentucky come up 
to me and say, “I was part of an Explorer School. I am teaching 
at a different school now, but I can tell you I changed the way I 
teach because of that experience. I am teaching more real world, 
using NASA content, with my students today, so they see where 
they are going to apply their science and math.” She said it has 
made such a difference in the responsiveness and interest of her 
students in STEM. 

Now, we certainly use our Digital Learning Network, which is an 
opportunity for us to connect schools to our engineers and scientists 
at our centers. That is really what NASA has to add, our content 
that is new and relevant, and information that may or probably 
isn’t in a textbook, our facilities and our experts and our people. 
So our Digital Learning Networks, for instance, I saw a school from 
the State of Washington in a dialogue with the scientists at John- 
son Space Center. So through that vehicle, we can reach every 
school in every state, even if they are not within a radius where 
they can travel to one of our NASA centers, although we certainly 
encourage that. 

We balance our portfolio, looking at higher education, under- 
graduate, graduate, and the K-12 as a really important part of our 
continuum, to have a pipeline to our workforce, not only for NASA, 
but also our contractors. And informal education is certainly a way 
to engage the public at large. We look forward to more opportuni- 
ties to work with our counterparts in other agencies, educators, to 
really make sure we are on target meeting the needs of educators 
and students today. 

Thank you. 

[The prepared statement of Dr. Winterton follows:] 
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Prepared Statement of Joyce L. Winterton 

Chairman Baird and Members of the Subcommittee, thank you for the oppor- 
tunity to appear before the Subcommittee today to discuss NASA activities that sup- 
port K-16 science, technology, engineering, and mathematics (STEM) educational 
programs. 

NASA recognizes the important role education plays in developing the diverse sci- 
entific and technological workforce required to advance this Nation’s economic lead- 
ership. The United States has a tremendous need to build, sustain, and deploy the 
skilled talent that will be required to continue America’s preeminence in space and 
aeronautics research and development in the coming decades. NASA serves as a 
contributor for achieving such goals. 

To ensure our future explorers will be ready to continue the journey, NASA is 
working with one of its most vital partners — educators. This summer, NASA will ig- 
nite the flame of knowledge with the first space flight of one of NASA’s most famous 
educator. Mission Specialist and Educator Barbara Morgan will engage students 
and educators worldwide from 240 miles above Earth aboard the International 
Space Station. 

NASA Administrator Michael Griffin recently stated, “The greatest contribution 
that NASA makes in educating the next generation of Americans is by providing 
worthy endeavors for which students will be inspired to study difficult subjects like 
math, science and engineering because they too share the dream of exploring the 
cosmos.” 

To this end, NASA educational investments are designed to: 

1. Strengthen NASA and the Nation’s future workforce — NASA will iden- 
tify and develop the critical skills and capabilities needed to ensure achieve- 
ment of exploration, science, and aeronautics. 

2. Attract and retain students in STEM disciplines through a progres- 
sion of educational opportunities for students, teachers, and fac- 
ulty — To compete effectively for the minds, imaginations, and career ambi- 
tions of America’s young people, NASA will focus on engaging and retaining 
students in STEM education programs to encourage their pursuit of edu- 
cational disciplines critical to NASA’s future engineering, scientific, and tech- 
nical missions. 

3. Engage Americans in NASA’s mission — NASA will build strategic part- 
nerships and linkages between STEM formal and informal education pro- 
viders. Through hands-on, interactive, educational activities, NASA will en- 
gage students, educators, families, the general public, and all agency stake- 
holders to increase America’s science and technology literacy. 

Experience has shown that exciting and compelling NASA missions truly can in- 
spire the next generation of explorers, innovators, and leaders. NASA’s unique pro- 
gram content, people, and facilities can be leveraged to spark interest, capture 
imaginations, and guide students toward careers in STEM fields while increasing 
their scientific and technologic literacy to the benefit of the Nation. 

To prepare future generations to manage and lead the cutting-edge research of 
tomorrow, strategic planning is essential. NASA has identified strategic goals and 
objectives that align its portfolio of education programs with the Human Capital Ini- 
tiatives under the President’s Management Agenda to build the workforce needed 
to meet core competences. All of NASA’s education efforts are part of an integrated 
Agency-wide approach to human capital management. 

NASA Education Programs support multiple goals and sub-goals in the 2006 
NASA Strategic Plan. Specifically, the education programs of the Agency contribute 
to the following outcomes: 

• Outcome 1: Contribute to the development of the STEM workforce in dis- 
ciplines needed to achieve NASA’s strategic goals through a portfolio of pro- 
grams. 

• Outcome 2: Attract students and retain them in STEM disciplines through a 
progression of educational opportunities for students, teachers, and faculty. 

• Outcome 3: Build strategic partnerships and linkages between STEM formal 
and informal education providers that promote STEM literacy and awareness 
of NASA’s mission. 

NASA Education Programs 

The manner in which the Agency will achieve these outcomes is detailed in the 
NASA Education Strategic Coordination Framework. The Framework was approved 
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by the NASA Strategic Management Council in 2006 and guides the planning, im- 
plementation, assessment and validation of the following portfolio of programs: 

The Higher Education Program focuses on supporting institutions of higher edu- 
cation in strengthening their research capabilities and providing opportunities that 
attract and prepare increasing numbers of students for NASA-related careers. The 
research conducted by the institutions contributes to the research needs of NASA’s 
Mission Directorates. 

The Minority University Research and Education Program (MUREP) en- 
gages under-represented populations through a wide variety of initiatives. Multi- 
year grants are awarded to engage minority institutions, faculty and students in re- 
search pertinent to NASA missions. The program focuses on retaining under-rep- 
resented and under-served students in STEM disciplines through completion of un- 
dergraduate or graduate degrees and entry into the scientific and technical work- 
force. 

The Elementary and Secondary Education Program provides K-12 educators 
with tools, experiences, and opportunities to further their education and participate 
in unique NASA learning experiences to enhance their knowledge of STEM and in- 
spire pursuit of STEM careers. The program supports the role of educational institu- 
tions, which provide the framework to unite students, families, and educators for 
educational improvement. 

Education Technology and Products (e-Education) sustains the research and 
development of technology applications, products, services and implementation of 
technology-enriched infrastructure in facilitating the appropriate and effective tech- 
nology based applications to enhance the educational process for formal and infor- 
mal education. In addition, e-Education identifies projects that will meet the objec- 
tive of the President’s Management Agenda to provide citizen-centric services re- 
lated to NASA Education efforts. 

The Informal Education Program is focused on increasing learning, educating 
students, educators and the general public on specific STEM content areas, and ex- 
panding the Nation’s future STEM workforce. Projects within the program produce 
supplemental educational materials that are standards based and designed to sup- 
port facilitators who are trained or qualified in STEM education fields, and are ac- 
tively working with participants to further enhance their understanding. Informal 
Education Programs also develop content based on educational standards and learn- 
ing objectives to supplement and enrich an experience, visual, or activity. 

The breadth of our portfolio, and how these programs have been implemented na- 
tionally, can be illustrated through the following examples: 

• Attracting students to the teaching profession, the NASA Educator Astro- 
naut project uses the visibility and educational opportunities created by the 
activities of the Educator Astronauts to inspire greater K-12 STEM achieve- 
ment, promote STEM careers, and elevate public esteem for the teaching pro- 
fession. In selecting Educator Astronauts, NASA identified and trained hun- 
dreds of our country’s top educators who are members of the Network of Edu- 
cator Astronaut Teachers (NEAT). Approximately 180 NEAT members are 
now in communities across America, each conducting workshops that reach 
about 90 educators per session. These efforts result in strengthening the 
STEM skills of approximately 10,000 teachers annually. 

• NASA Explorers Schools (NES) provide intensive training and on-site pro- 
fessional development to teachers in classrooms across the country. The NES 
project assists middle schools with improving teaching and learning in STEM 
education through professional development, stipends, grants, and curricular 
support based on NASA resources. In 2006, 5,339 teachers received intensive 
training as part of the NES project. Additionally, our Aerospace Education 
Services personnel conducted sessions across the Nation, reaching 13,938 edu- 
cators in other schools. 

• In addition to in-service workshops based on our missions, NASA is com- 
mitted to the pre-service training of our future educators. Through the Na- 
tional Pre-Service Teacher Conference, Pre-Service Teacher Institutes and 
Online Professional Development, NASA recruits STEM teachers to develop 
the confidence and skills to effectively teach mathematics and science using 
cutting-edge technology and educational materials. Such efforts have led to 
200 STEM-enhanced teachers instructing an average of 25 students per class- 
room for three years, impacting a projected total of 15,000 students. 
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• NASA’s four Mission Directorates provide opportunities for students to en- 
gage in NASA mission related experiences. For example, within NASA’s 
Science Mission Directorate, a broad spectrum of education activities are 
sponsored, ranging from kindergarten to post-graduate levels. All of NASA’s 
science missions and programs are required to have an education and public 
outreach component. Through a competitive, peer-review selection process, 
NASA provides funding dedicated to education and public outreach to re- 
searchers. NASA also sponsors graduate and post-doctoral fellowship opportu- 
nities. In addition, the Agency is looking for new ways to provide increased 
opportunities for students to gain greater experience developing and launch- 
ing their own science instruments, either in conjunction with science missions 
or through its suborbital rocket and balloon programs. 

• Launched in January 2006 as part of the New Horizons Mission, the Stu- 
dent Dust Counter is the first student-built instrument selected by NASA to 
fly on a planetary mission. Built by students at the University of Colorado 
at Boulder, the counter will monitor the density of dust grains in space. This 
data is of particular interest to researchers. Given the nine-year travel time, 
discoveries from this mission will engage today’s elementary school student 
until college when this spacecraft encounters Pluto. 

• Aeronomy of Ice in the Mesosphere (AIM) began its two-year mission on 
April 25, 2007, after a flawless ride to Earth orbit aboard an Orbital Sciences 
Pegasus XL rocket. AIM is the first mission dedicated to exploring mysterious 
ice clouds that dot the edge of space in Earth’s polar regions. With AIM, 
Hampton University in Virginia has become the first Historically Black Col- 
lege and University to lead a NASA satellite mission. Undergraduate and 
graduate students from various STEM disciplines will have an opportunity to 
join faculty researchers in the analysis of collected data. 

• In February 17, 2007, NASA launched five Time History of Events and 
Macroscale Interactions during Substorms (THEMIS) microsatellites to 
study the Earth’s magnetosphere. THEMIS will help scientists understand 
how and why space storms create havoc on satellites, power grids, and com- 
munication systems. Students will work with scientists to unravel a variety 
of scientific mysteries. 

• NASA’s support of higher education students is embodied by the National 
Space Grant College and Fellowship Program, which continues to pro- 
vide fellowships and scholarships to students across the country. Recent sta- 
tistics show that, of the pool of students who completed their degrees, 31 per- 
cent were employed in STEM careers and 48 percent continued their edu- 
cation to the Master’s, Ph.D., or post-doctoral levels. Many consortia have im- 
plemented hands-on, university student-led projects in aeronautics, rocketry, 
scientific ballooning, rocketry, and nano- and micro-satellite development. 
These types of projects provide the professional training that enable students 
to be fully prepared to enter the STEM workforce. 

Portfolio Management Process 

Such a diverse portfolio requires effective management of the Agency’s education 
portfolio both internally and externally, with clear roles and responsibilities. As the 
Assistant Administrator for Education, I am responsible for ensuring that the Edu- 
cation Outcomes as reflected in the 2006 NASA Strategic Plan are achieved. I serve 
as both the head of the Office for Education, managing all responsibilities assigned 
to the Office and also as the Chair of the Education Coordinating Committee (ECC), 
ensuring the overall planning, coordination, and integration of the Agency’s entire 
education portfolio. 

NASA’s ECC is a collaborative structure that maximizes NASA’s ability to main- 
tain an integrated education portfolio and strategically manage the implementation 
of numerous programs, projects and activities in a distributed system. The com- 
mittee consists of representatives of the Agency’s Office of Education, the four Mis- 
sion Directorates that provide mission related content, and the ten NASA Center 
Education Offices, among others. The committee develops education strategy and 
supports me in coordinating education efforts throughout the Agency. The ECC also 
provides checks and balances for effective internal control and ensures the success- 
ful achievement of education goals and portfolio effectiveness. 

Collaboration and Coordination with Other Federal Agencies 

NASA’s Office of Education is continually engaged in collaboration with other fed- 
eral agencies, including: the Department of Education, National Science Foundation, 
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Federal Aviation Administration, Department of Commerce, Smithsonian Institu- 
tion, Department of the Interior, and Department of Energy. Additionally, NASA 
collaborates with state STEM education coalitions, through the National Alliance of 
State Science and Mathematics Coalitions, the District of Columbia, Puerto Rico, 
and the U.S. territories. Each of our Centers works closely with State and local de- 
partments of education to ensure that our resources are tailored to support the 
needs of the education community. We have worked hard to ensure that we under- 
stand and can respond to the needs of State or local districts. 

Collaboration and coordination also occur in a number of forums in the Federal 
Government to ensure that NASA’s activities in K-16 STEM education are com- 
plementary and not redundant with the programs of other federal agencies. Addi- 
tionally, NASA has actively participated in the Congressionally-mandated Academic 
Competitiveness Council (ACC), which found there is a dearth of evidence of effec- 
tive practices and activities in STEM education and made recommendations to inte- 
grate and coordinate federal STEM programs. 

In February 2007, NASA and the National Science Foundation (NSF) signed an 
historic agreement to work together and coordinate efforts to expand opportunities 
for promoting STEM education and to broaden the participation of the under-rep- 
resented in those areas. The Memorandum of Understanding (MOU) between NASA 
and NSF promotes a comprehensive knowledge base to be shared between the agen- 
cies to address national challenges and manage the agencies’ resources more effec- 
tively. It reflects the goals of the Administration’s American Competitiveness Initia- 
tive, whose cornerstone is a commitment to increase investments in basic research 
in the physical sciences and engineering, strengthen K-12 math and science edu- 
cation, and build a well-educated, skilled workforce. One of the first results of the 
collaboration was a three day joint NASA-NSF Research Education Opportunity 
Conference for Principal Investigators, Faculty, and Partners. Over three hundred 
members of the academic community gathered to be trained on ways to strengthen 
their ability to compete for research grants and to leverage their partnerships with 
the agencies. 

Earlier this month, NASA and the Federal Aviation Administration (FAA) signed 
an MOU to foster the development of students’ skills in STEM. The agreement sup- 
ports the FAA’s mission to provide the safest, most efficient airspace system in the 
world and NASA’s mission to pioneer the future in space exploration, scientific dis- 
covery and aeronautics research. The partnership includes a broad range of coopera- 
tive outreach activities. The agencies’ initial focus is on a NASA resource called, 
“Smart Skies.” Smart Skies is an online air traffic control simulator for students in 
fifth through ninth grades. It offers a fun and exciting way to learn math and skills 
central to air traffic control while providing multiple modes of problem solving for 
students who learn in different ways. The agreement unites the strengths of both 
agencies to provide the best of aviation-related educational products and experiences 
to the widest possible population of students and educators. 

Sharing Best Practices 

Through our work with the ACC, we are strengthening our evaluation methodolo- 
gies and sharing some of our best practices, for example: 

The Harriett G. Jenkins Pre-doctoral Fellowship Program (JPFP) is a model 
of a STEM education pipeline program that can be replicated by other agencies as 
a best practice. While the success of the JPFP can easily be quantified by counting 
the number of students participants (121), the number of awards provided to con- 
duct research at a NASA center (90) or the number of successful mentoring relation- 
ships that were established through this program (121), the greatest accomplish- 
ment of the program is an exceptionally diverse group of under-represented STEM 
scholars who are excited about pursuing NASA-related advanced degrees that will 
equip them to participate in the space exploration workforce. To date, the Jenkins 
project has produced 34 M.S. degrees and 32 Ph.D. degrees in NASA-related dis- 
ciplines. 

Another stellar NASA project identified as a best practice is the Science, Engi- 
neering, Mathematics, and Aerospace Academy (SEMAA). The Ash Institute 
for Democratic Governance and Innovation at Harvard University’s John F. Ken- 
nedy School of Government announced that SEMAA was among the top eighteen 
programs in the 2007 Innovations in the American Government Awards competi- 
tion. Selected from a pool of nearly 1,000 applicants, these initiatives are being rec- 
ognized as the government’s best efforts for their novelty and creativity, effective- 
ness at addressing significant issues, and potential to be replicated by other jurisdic- 
tions. Finalists presented before the National Selection Committee at the Kennedy 
School on May 15, 2007, and winners will be announced in September. 
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Evaluation of NASA Education Programs 

The Agency’s many Education initiatives have not been evaluated in a comprehen- 
sive, rigorous manner to indicate how well all of our programs are performing in 
support of our outcome goals. We are committed, however, to enhancing and improv- 
ing our evaluation procedures. 

The Agency has taken several major steps to improve the evaluation function by: 

(a) incorporating a detailed evaluation plan into its Education Strategy Frame- 
work; 

(b) defining an enhanced set of outcome-based performance measures; articu- 
lating specific roles and responsibilities to ensure accountability; and, 

(c) allocating the resources necessary to support rigorous evaluations and the 
overall evaluation function. 

A range of processes will be used to capture the total picture of education across 
NASA and to assess the education portfolio for its effectiveness in: achieving the 
stated outcomes; establishing linkages within the framework; and determining the 
level of quality, impact and comprehensiveness of the portfolio. The ECC will em- 
ploy an appropriate mix of methodologies, ranging from basic quantitative data to 
qualitative information, to assess the overall condition of the education portfolio. 

Coincident with the adoption of a new education framework and outcomes in 
FY06, NASA developed a corresponding set of objectives and outcome measures. 
Baselines for these measures are being established with FY07 data. The outcome 
measures include, but are not limited to the following: 

• Percentage of student participants employed by NASA, aerospace contractors, 
universities, & other educational institutions. 

• Percentage of undergraduate students who move on to advanced education in 
NASA-related disciplines. 

• Level of student interest in science and technology careers resulting from ele- 
mentary and secondary NASA education programs. 

The most significant improvement NASA is making to its evaluation efforts is to 
make use of independent, credible evaluators to measure the effectiveness of edu- 
cation investments. Project-level evaluations will be conducted on three to five of our 
major projects each year, with the objective of evaluating each project at least once 
every five years. In collaboration with the National Science Foundation and the Of- 
fice of Management and Budget, we are working to determine the best ways to 
apply a Randomized Controlled Trial (RCT) model of evaluation to demonstrate the 
impact of our portfolio of programs. Projects that cannot be reliably evaluated using 
RCT methods will be evaluated in an objective and credible manner, conforming to 
the standards of professional practices. 

Public Awareness and Access to NASA Education Programs 

NASA Education is a cross-cutting process that engages the public in shaping and 
sharing the experience of exploration and discovery. The President’s FY 2008 budget 
request for NASA’s Education program is $153.7 million. Through the Office of Stra- 
tegic Communications, the Agency is building and maintaining public awareness for 
the activities and goals focusing on science, education, aeronautical research and ex- 
ploration. 

As part of the Agency’s long-term strategy in promoting public awareness, Na- 
tional Education Campaigns designed to build a comprehensive education initiative 
that engage diverse audiences with tailored modes of interaction have become com- 
mon practice for assisting the Agency with public engagement and the formation of 
national and international visibility and recognition. STS-118, the first Space Flight 
of an Educator Astronaut, is a good example of a National Education Campaign de- 
signed not only to engage students and educators but also increase America’s 
science and technology literacy. 

NASA disseminates its education content including STEM-related materials 
through resources designed to reach all education audiences — formal, informal and 
the public at-large — as well internal dissemination networks such as Aerospace 
Education Services Program (AESP) and Space Grant. 

The NASA Portal opens the door to all the resources that NASA has available. 
From there, educators can either download materials for use, or obtain copies from 
the Central Operations of Resources for Educators (CORE). CORE is a worldwide 
distribution center for NASA’s educational material. 
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The Role of Partnerships 

Strategic alliances with non-governmental organizations provide an immediate 
springboard as unfunded collaborators to produce, market, and distribute edu- 
cational information about NASA’s projects and programs. NASA’s partnership with 
the International Technology Education Association is one of many venues the 
Agency uses to reach students and educators across the country. Other organiza- 
tions include the National Science Foundation, National Institute of Aerospace, Na- 
tional Science Teachers Association, AOL’s Kids On-Line, the Girl Scouts of the 
USA, Imaginary Lines, and Reader’s Digest. 

Imagine, with the right partners, what NASA can do to strengthen and support 
STEM education. Powerful technologies can enable new learning environments 
using simulations, visualizations, immersive environments, game playing, intelligent 
tutors and avatars, learner networking, and usable building blocks of content. These 
capabilities can create rich and compelling learning opportunities that meet the 
needs of learners while empowering educators to unlock the potential in each stu- 
dent’s heart and mind. NASA can unite with the technology and education commu- 
nities in dialogue, understanding and action. Students and educators can have ac- 
cess to a new renaissance of learning for the benefit of the Nation and the world. 

Conclusion 

I would like to commend the Subcommittee for its efforts to improve K-16 STEM 
education. The educational achievement of America’s next generation is an issue 
that reaches our nation at all levels. NASA will continue to partner with federal, 
industry, State and local organizations and invest our resources toward a shared vi- 
sion to secure those jobs critical to the 21st century workforce. This means not only 
inspiring the next generation of leaders and explorers but also providing educators 
with unique resources to support educational excellence in STEM while improving 
scientific literacy. 

The President, Administrator Griffin, and all of NASA share the belief that a 
highly educated and well-prepared workforce has been and continues to be essential 
to this country and the Agency. NASA’s investment in education is indeed an invest- 
ment in America’s future. 

Thank you for the opportunity to participate in this important hearing. I am pre- 
pared to respond to any questions you may have. 

Biography for Joyce L. Winterton 

Dr. Joyce Leavitt Winterton, NASA’s Assistant Administrator for Education, di- 
rects the development and implementation of the Agency’s education programs that 
strengthen student involvement and public awareness of its scientific goals and mis- 
sions. In this role, she leads the agency in inspiring interest in science, technology, 
engineering, and mathematics, as few other organizations can through its unique 
mission, workforce, facilities, research and innovations. As Assistant Administrator 
for Education, Winterton chairs the Education Coordinating Committee, an agency- 
wide collaborative structure that maximizes NASA’s ability to manage and imple- 
ment its education portfolio. The ECC works to ensure that the Agency’s education 
investments are focused on supporting the Nation’s education efforts to develop the 
skilled workforce necessary to achieve the Agency’s goals and objectives. Before com- 
ing to NASA, Winterton served as the Director of Education Programs for USA 
TODAY, and developed educational strategies, resources and partnerships for its K- 
12 and collegiate programs. During her nine years at USA TODAY, she created in- 
novative cross-curricular educational approaches, including case studies, content de- 
velopment and on-line collaborations. She was the founder and President of 
Winterton Associates, a consulting firm that specializes in working on joint projects 
with business and industry, education, and government. The firm has served as the 
evaluator for National Science Foundation projects and U.S. Department of Edu- 
cation-funded programs, including six national skill standards projects since 1991. 
Winterton’s previous experience includes serving as the team leader for partner de- 
velopment for the National Future Farmers of America student organization, where 
she planned and developed partnerships and strategies to communicate the benefits 
of agricultural education and a student organization with over 450,000 members. 
She has also been an education training consultant for FranklinCovey Inc. where 
she facilitated time management and personal effectiveness workshops for national 
student leadership organizations. In 1986, Winterton became the executive director 
of the National Council on Vocational Technical Education, a Presidential Advisory 
Council providing recommendations to the President, Congress and the Secretary of 
Education. Additionally, Winterton served as the deputy assistant secretary for vo- 
cational and adult education in the United States Department of Education and was 
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the first Director of the Presidential Academic Fitness Awards program. She also 
was a professional staff member for the U.S. Senate Committee on Labor and 
Human Resources. She has served on a number of national education boards and 
advisory panels. Winterton has been a high school teacher, a teacher educator and 
a home economist in business. She received the Lawrence Prakken Professional Co- 
operation award from the International Technology Education Association and was 
recognized as an outstanding alumna from Colorado State University and also the 
Family, Career and Community Leaders of America. She earned her Bachelor’s and 
Master’s degrees in home economics education from Utah State University in Logan. 
In 1978, she completed her doctorate in teacher education and administration at 
Colorado State University in Fort Collins. 

Mr. McNerney. Thank you, Dr. Winterton. Mr. Valdez. 

STATEMENT OF MR. WILLIAM J. VALDEZ, DIRECTOR, OFFICE 

OF WORKFORCE DEVELOPMENT FOR TEACHERS AND SCI- 
ENTISTS, OFFICE OF SCIENCE, DEPARTMENT OF ENERGY 

Mr. Valdez. Thank you. Mr. Chairman and Mr. Ehlers, thank 
you for inviting me to testify at this important hearing. 

I have submitted written testimony that makes three points. 
First, federal S&T mission agencies, such as the Department of En- 
ergy, have enormous resources that could be devoted to STEM edu- 
cation and workforce development. Those resources naturally com- 
plement what is offered by the Department of Education and the 
National Science Foundation. We are developing partnerships with 
other federal agencies, NSF, the Department of Education, and 
other organizations that have a strong interest in STEM education, 
as a way to leverage our resources within the Department of En- 
ergy. 

Second, the Office I manage, Workforce Development for Teach- 
ers and Scientists, has a great deal of work to do in the area of 
program evaluation. We are in the process of developing a com- 
prehensive and rigorous evaluation program that builds upon the 
recommendations of the Academic Competitiveness Council, and 
what experts told this committee at the May 15 hearing. 

Third, my Office is engaged in an extensive planning process 
that is leading to a reprioritization of our programs to meet today’s 
challenges in STEM education and workforce development. We are 
now developing business plans for all of our programs that describe 
their goals, resource requirements, and connection to the DOE mis- 
sion. 

These three points are discussed in more detail in my written 
testimony, and I welcome an opportunity to answer any questions 
you might have about them. But I would like to take a moment to 
discuss one of the questions that you asked in your May 29 letter 
of invitation to this hearing. What do you recommend as the most 
effective role your agency can play in improving STEM literacy? 

I have had literally dozens of conversations with experts in the 
field on this question, including most of my colleagues at this table, 
and just in the past two weeks with Frank Owens of the National 
Science Teachers Association, Sally Shuler of the National Science 
Resource Center, and Iris Weiss of Horizon Research, who testified 
at this hearing on May 15. The conversations I had with these ex- 
perts, who I consider to be among the thought leaders in STEM 
education in the United States, have led me to the following obser- 
vations. 
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All three agreed that DOE programs, which emphasize experien- 
tial learning, hands-on opportunities for students and educators, 
and rigorously designed programs at our National Laboratories, fill 
a critical void in STEM education and workforce development. Iris 
Weiss, for example, said that DOE needs to provide an authentic 
research experience well beyond the brief cookbook experience that 
students and educators tend to get even at the undergraduate 
level. Students and educators learn by doing. We provide them 
with an opportunity at our National Laboratories to learn with 
some of the best mentor scientists in the world, but, and this is a 
very big caution, our programs must be properly designed to maxi- 
mize their effectiveness. This is why we are developing business 
plans for all of our programs, and will have those plans reviewed 
by outside experts. Our program design must be open, transparent, 
and conform with the best standards known to the STEM edu- 
cation community. Iris Weiss asked me if we really would take 
their suggestions and criticisms to heart, and my answer was, “Of 
course.” 

All three experts were firm in their belief that we must find a 
way to sustain our programs, and link them to what matters to 
educators and students. Frank Owens suggested that DOE partner 
with NSTA to help develop voluntary national science certification 
standards for educators. This would take two forms: first, making 
a structured laboratory research experience part of the voluntary 
national certification standards that NSTA is developing; and sec- 
ond, utilizing DOE’s world-class scientific talent to partner with 
NSTA to develop their online science content modules. 

NSTA will propose a partnership structure to accomplish this 
with the Department of Energy. Iris Weiss suggested that we also 
work with the Council of State Science Supervisors and other 
groups, which we will do. Finally, every expert we have spoken 
with has said that we must carefully evaluate our programs. Iris 
Weiss and Sally Shuler, who really know this business inside and 
out, agree that we currently are only able to measure improve- 
ments to content knowledge and interest in pursuing a science ca- 
reer after participation in the kinds of experiential learning pro- 
grams that we manage, but we currently cannot measure whether 
those increases lead to improved test scores and an ability to per- 
form science. I am very interested in talking with NSF and my col- 
leagues at the table and other experts about ways to fill these gaps 
in knowledge and improve our program evaluation. 

Overall, all three experts agree that federal S&T agencies need 
to do a better job of talking with one another, sharing best prac- 
tices, and leveraging resources. They expressed enormous frustra- 
tion that we don’t have a “one-stop shopping” resource for pro- 
grams, evaluation techniques, and outreach. Even the simple 
things, such as a common application for K-12 educators who want 
to enter into a research experience, would be a big help, they said. 
The series of hearings that this committee is holding should help 
in that regard. 

I would like to thank you, Mr. Chairman, and other Members of 
the Committee, for investing your time and energy into studying 
this challenge. I look forward to working with you and your staff, 
and answering any questions you might have. 
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[The prepared statement of Mr. Valdez follows:] 

Prepared Statement of William J. Valdez 

Thank you, Mr. Chairman and Members of the Subcommittee, for this opportunity 
to discuss the role that the Department of Energy’s (DOE) Office of Science plays 
in scientific and technical workforce development and education. We appreciate your 
strong commitment to improving science and math education and training in the 
United States. 

The Office of Science is the Federal Government’s largest supporter of civilian 
basic research in the physical sciences. This basic research supports the Depart- 
ment’s missions in energy, the environment, and national security. The Office of 
Science manages 10 national laboratories and more than 30 major scientific user fa- 
cilities that provide the scientific community with state-of-the-art research tools that 
help accomplish the Department’s goals and maintain U.S. competitiveness in 
science and technology. 

The Department’s most significant contribution to the development of a scientific 
and technical workforce has been through the support of graduate students pur- 
suing advanced degrees, post-doctoral students who work on research projects, and, 
to a much smaller degree, hands-on research opportunities for undergraduate stu- 
dents and K-12 educators and informal experiential learning opportunities for K- 
12 students. These individuals utilize DOE research facilities and work side-by-side 
with the scientific and technical staff at the national laboratories. 

Those national laboratories are unique settings for research, mentoring, and col- 
laboration. Through structured and unstructured workforce development and 
science education programs at DOE’s 17 national laboratories, the power authorities 
and other DOE facilities, the Department engages with more than 250,000 students 
of all ages and 19,000 K-12 educators on an annual basis. 

The Office of Workforce Development for Teachers and Scientists (WDTS), which 
I manage, is the only program office in DOE that has a specific mission in science, 
technology, engineering and mathematics (STEM) workforce development and edu- 
cation. Our programs reach 600 undergraduate students, 16,000 K-12 students, and 
150 K-12 educators annually. We do this with an $8 million annual budget and 
under specific statutory authority (Public Laws 93-438 and 101-510, and most re- 
cently the Energy Policy Act of 2005). 

The 17 DOE national laboratories, the power authorities and other DOE facilities 
use WDTS funding as “seed money” to develop complementary programs that are 
designed to meet their local needs. Our programmatic philosophy is “nationally de- 
signed programs, locally delivered.” This model has relied on partnerships within 
the Department of Energy and with external organizations. 

While WDTS directly funds 600 undergraduate students for summer internships, 
the total number of undergraduate research interns at all of the DOE laboratories 
is 4,100. Similarly, WDTS directly funds 150 K-12 educators, but a total of 19,300 
K-12 educators are involved in programs at DOE laboratories and facilities. 

Partnerships enable WDTS to coordinate with and leverage the resources and ca- 
pacity of the Office of Science (SC). SC works with more than 300 of the top univer- 
sities in the Nation, manages 10 of the biggest national laboratories in the Federal 
Government, and deals directly with hundreds of high technology companies. 

In recognition of widespread concern about STEM workforce development, the 
Secretary of Energy, in 2006, commissioned a review by the Secretary of Energy Ad- 
visory Board of the Department’s activities in STEM education. That review con- 
cluded that DOE has a clear role in STEM education and that partnerships are the 
primary vehicle we should use to achieve our goals. The Board stated: 

“[A] review of the Department’s educational programs as well as a review of the 
educational efforts in other federal agencies, leads us to our conclusion that 
DOE has a significant opportunity to enhance STEM education in the Nation. 
Moreover, it is clear from our review (as well as from the GAO reports) that 
the educational activities of DOE and other federal agencies could benefit from 
increased cooperative activities with one another, with industry, with colleges 
and universities, and with science teachers’ professional organizations. In both 
nationwide influence and in cooperative partnerships, DOE is already posi- 
tioned to take a leadership role. DOE’s national laboratories are geographically 
distributed over the country, allowing access to teachers across the Nation. 
Moreover, the network of national laboratories is also tightly linked with indus- 
trial and academic resources, giving DOE the ability to forge educational part- 
nerships that can extend its reach, and therefore also its capacity to enhance 
STEM education nationwide.” 
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As a result of its stakeholder meetings and other outreach efforts, WDTS has had 
discussions with a wide range of organizations proposing partnerships. Let me give 
you four examples of partnerships that could make our programs more effective: 

• WDTS has an existing partnership with the National Science Foundation 
(NSF) that illustrates how federal resources can be effectively leveraged. DOE 
has the 17 national laboratories, but NSF has greater access to under- 
graduate and educator populations. Our agreement with NSF enables us to 
share programs, with a result that in FY 2006 NSF supported 195 educators 
and students at seven of our national laboratories. (Table 1) 
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This is a beginning, but we could do more. DOE mentor scientists who par- 
ticipate in Office of Science and other DOE programs have a long history of 
working with students, and many have indicated they are eager to expand 
their efforts. One resource that could help is the federal laboratory system. 
The Federal Government owns more than 250 national laboratories across 
the Nation, and many of these have STEM workforce needs similar to those 
of DOE. Thus, WDTS could partner with USDA labs, for example, to prepare 
the future workforce to support the expanding bio-fuels industry, or with De- 
partment of Defense laboratories to develop our national security workforce. 

• WDTS is engaged in extensive discussions with the Department of Education 
on better support for the Administration’s Adjunct Teacher Corps initiative. 
The mentor scientists at DOE’s national laboratories could constitute a prom- 
ising potential core of the Adjunct Teacher Corps. Sandia National Labora- 
tories, which is part of the National Nuclear Security Administration within 
DOE, is taking a leadership role with WDTS to structure a program that 
would enable us to work with the Department of Education to achieve the Ad- 
ministration’s goal of placing 30,000 adjunct teachers in the Nation’s class- 
rooms by 2015. If Sandia’s pilot program with the Department of Education 
is successful, the concept could be expanded to other federal agencies with na- 
tional laboratories and pools of mentor scientists. 

Secretary of Education Margaret Spellings on May 9, 2007 commented on this 
emerging partnership: “When I was in Senator [Jeff] Bingaman’s state of 
New Mexico I visited a local high school where scientists from Sandia Labs 
were teaching chemistry. We need to make this the norm around the coun- 
try.” 

• The core element of WDTS’s programs and other programs carried out at the 
DOE laboratories is providing educators and students with hands-on research 
experience. These research experiences supplement what students learn in 
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the classroom and help educators better understand the process of science. 
Thus, we want to partner with organizations like universities and corporate 
laboratories that have similar infrastructure to that of the DOE national lab- 
oratories. As a start, we have entered into discussions with a university and 
a major non-profit science educational group in Boston about pilot programs 
that would share resources and capabilities. 

• WDTS is developing what we are calling a “trusted partners” approach to 
reach under-represented populations. Students and educators tend to learn 
about our programs primarily through recommendations from individuals and 
organizations whose opinions the students and educators themselves trust. 
This is particularly true of students and educators from under-represented 
populations who have not built a trust relationship with the Department of 
Energy. As a result, we are exploring partnerships with several national orga- 
nizations to help identify their most promising students and educators for our 
programs. We have had discussions, for example, with a major Hispanic com- 
munications network about developing innovative approaches that reach the 
best and brightest Hispanic students and teachers for our programs. 

Evaluation 

I would now like to turn to the need for evaluation and intelligent program de- 
sign. 

In this regard, I would like to commend the Department of Education and the Of- 
fice of Management and Budget (OMB) for the work they have done through the 
Academic Competitiveness Council (ACC). The catalogue of existing STEM edu- 
cation programs in the Federal Government and the emphasis the ACC Report 
places on the need for rigorous evaluation catalyzed a discussion in Washington, 
D.C. policy circles about the need for rigorous evaluation of STEM education and 
workforce programs. 

The ACC Report’s recommendations were influential in the development of 
WDTS’s future direction. This was a discussion that was much needed because, 
frankly, WDTS has done a poor job over the past 10 years of rigorously evaluating 
our programs. As a result, and under the specific direction of Under Secretary for 
Science Raymond Orbach, we are committed to improving our ability to evaluate the 
impact and effectiveness of our programs. 

We have data that indicate our experiential learning programs are yielding good 
results (i.e., promoting interest in STEM fields). We are in the process of developing 
a plan for more rigorous study of the program that will enable us to demonstrate 
the program’s impact. Based on the results, we will be able to refine the program 
and pursue the most effective strategies going forward. 

One lesson that was reinforced by the ACC process is that evaluation and assess- 
ment are crucial to the effective design of STEM workforce programs. For example, 
during the 1980s and 1990s, the Department funded rigorous longitudinal workforce 
studies that enabled program managers to identify specific future workforce needs. 
WDTS is in the process of re-invigorating that effort and within the next 12 months 
will have completed a pilot workforce study that identifies the workforce needs, by 
scientific discipline, for the Office of Science federal and national laboratory staff. 

This workforce study is being done in collaboration with the National Nuclear Se- 
curity Administration (NNSA). By including the 10 national laboratories managed 
by the Office of Science and the three defense national laboratories managed by 
NNSA (Lawrence Livermore National Laboratory, Sandia National Laboratories, 
and Los Alamos National Laboratory) in the study, we will account for the majority 
of the R&D performed by the Department. In future years, we hope to include other 
DOE laboratories and R&D programs in this effort. 

Identifying gaps 

Rigorous evaluation of programs and the use of workforce data from the analysis 
that we will do over the next 12 months will enable WDTS to identify opportunities 
to improve our STEM workforce development and education efforts. 

WDTS currently manages nine programs for students and educators. Those pro- 
grams emphasize experiential learning opportunities for students and educators, 
such as the Academies Creating Teacher Scientists (ACTS) program; and world 
class celebrations of scientific achievement for students, such as the National 
Science Bowl. We have two decades of experience managing these types of programs 
and believe that they are effective and are contributing to our nation’s efforts to im- 
prove STEM education and proficiency, although more rigorous evaluations are in 
order. 
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But conversations with our stakeholders and our own internal analysis have re- 
vealed that there are opportunities for our programs to better achieve their objec- 
tives. Let me give you two examples: 

• One potential gap is in the development of talent in our federal workforce. 
While one of our goals is to encourage students to join federal service, we do 
not have programs in place that provide a clear link for them to seek employ- 
ment with the Department, such as helping them navigate the difficult fed- 
eral hiring process. In addition, we do not have the workforce assessment 
tools in place that would inform us about whether we need more physicists, 
chemists, or engineers. The workforce assessment we are doing will help in 
that regard. We are also working with DOE’s Chief Human Capital Officer, 
Dr. Jeff Pon, to develop programs specifically targeted at the federal STEM 
workforce. 

• Another area for improvement is our collective need to better align agency 
STEM efforts with larger federal mission needs. Representatives from various 
federal agencies have emphasized the need to work collaboratively to solve 
our mutual STEM education and workforce challenges. One result is that my 
colleague, Dr. Joyce Winterton of NASA, has taken the initiative to form a 
brown bag lunch group of federal science and technology agencies as a forum 
for discussion and collaboration. I am also talking with the Federal Labora- 
tory Consortium, the Triangle Coalition and a host of other groups about 
partnerships designed to bring federal agencies together with the educational 
community and industry. We need to work together and, in fact, a grassroots 
process supported by federal S&T agencies has already begun. 

Conclusion: 

I would like to conclude by highlighting several statistics: 

• $135 billion — the annual federal investment in R&D that is managed by 34 
agencies 

• 257 — federal laboratories that belong to the Federal Laboratory Consortium 
and are active in communities nationwide 

• 206,000 — federal scientists and engineers (not including contractors) 

When all of these numbers are put together, it is evident that we have excellent 
resources for a coordinated federal response to the Nation’s STEM education chal- 
lenge. 

The 34 federal R&D mission agencies — such as NASA, NOAA, DOD, NIH, USDA 
and EPA — have a long-term and enduring interest in their workforces and STEM 
education. This has been a strong federal resource that can continue to support our 
efforts to address the national challenge of educating the future U.S. workforce and 
helping to prepare our citizens for the emerging era of scientific discovery and inno- 
vation. 

Thank you for offering me this opportunity to provide a perspective on this impor- 
tant issue. I look forward to answering the Committee’s questions. 

Biography for William J. Valdez 

Bill Valdez is the Director of the Office of Workforce Development for Teachers 
and Scientists within the Department of Energy’s Office of Science. His responsibil- 
ities include developing workforce strategies for the Department’s scientific and 
technical workforce, and creating opportunities for students and educators to par- 
ticipate in the Nation’s research enterprise as a means to improving the competi- 
tiveness of U.S. industry and overall scientific literacy. 

In addition, Mr. Valdez has been leading an interagency effort, coordinated by the 
White House Office of Science and Technology Policy, that is designed to establish 
credible outcome measures for basic research, create new evaluation methods that 
focus on systems level analysis, and promote business models that will enable fed- 
eral R&D managers to improve investment decisions. 

Previously, Mr. Valdez was the Director of Planning and Analysis at the Depart- 
ment of Energy’s Office of Science. His responsibilities included corporate strategic 
planning, R&D evaluation, and federal S&T policy development. 

Mr. Valdez was elected as a Fellow of the American Association for the Advance- 
ment of Science in 2006 and is Vice Chair of the Senior Executive Association’s 
Board of Directors. He was elected to the Board of Directors of the Senior Executive 
Association in 2005. 

Mr. Valdez has held various positions at the Department of Energy since 1994, 
including serving as Executive Director of the DOE R&D Council and developing 
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evaluation techniques for technology transfer programs. Mr. Valdez also served at 
the White House Office of Science and Technology Policy from 1998-99. His respon- 
sibilities included co-authoring a report on strategies designed to improve the future 
scientific workforce as the Nation’s demographics change, developing interagency 
technology initiatives, and advising on international energy initiatives. 

Prior to working at DOE, Mr. Valdez worked as a Senior Project Manager in pri- 
vate industry where he provided strategic planning services to Asian and European 
multi-national corporations. 

Mr. Valdez received a Bachelor of Arts from the University of Texas and his Mas- 
ter of Arts in International Economics and Energy Policy from the Johns Hopkins 
School of Advanced International Studies. 

Mr. McNerney. Thank you, Mr. Valdez. Dr. Fuchs. 

STATEMENT OF DR. BRUCE A. FUCHS, DIRECTOR, OFFICE OF 

SCIENCE EDUCATION, NATIONAL INSTITUTES OF HEALTH 

Dr. Fuchs. Thank you, Chairman McNerney and Congressman 
Ehlers. I want to thank you for this invitation to appear before you 
to discuss some of NIH’s STEM programs. It is an honor for me to 
appear before this subcommittee that has worked so hard to im- 
prove STEM education in this nation. 

I will apologize for the PowerPoint. I am an old professor, and 
old habits are hard to break. I would like to briefly discuss some 
of the things that I believe federal agencies can do to help with 
STEM education in this country. First, we can partner with outside 
agencies and experts to design exemplary model programs for a va- 
riety of things, instructional materials, teacher professional devel- 
opment, and then rigorously evaluate those programs. It is not that 
every agency should get involved in each activity, but I think every 
agency can select something to do well. We need to know what 
works, what doesn’t, and most importantly, why. This kind of in- 
depth design and evaluation research is unlikely to come either 
from states or private industry working alone. 

Second, I believe that the responsible federal agencies, and this 
is primarily NSF, Department of Education, and NIH through its 
National Institute on Child Health and Development, need to sup- 
port high-quality, scientifically based education research. Unfortu- 
nately, we have scientific evidence to support only a small number 
of items related to math and science teaching that we didn’t know 
25 years ago. We must not be in the same situation 25 years from 
now. 

Lastly, I would like to suggest that federal scientists, because of 
their many interactions with students and scientists from around 
the world, have an important insight into what it takes to compete 
in today’s world. This insight should be used to help define the 
world-class standards to which our schools must aspire. Sadly, 
many of our states’ science and math standards cannot presently 
be considered to be world-class. 

I would like to tell you briefly about two programs that we have 
at the NIH, and illustrate how they are working with educators in 
the states. The first is a grants program known as the Science Edu- 
cation Partnership Award. This is a peer-reviewed program located 
within NIH’s National Center for Research Resources. These grants 
are used to establish partnerships within a community to enhance 
the teaching of science within that community. For example, a 
partnership might be developed between a university and a science 
museum, or a university and a school district. There are currently 
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70 active SEPA grants in 39 states, a number of which focus on 
underserved populations within those communities. 

The second program I would like to highlight is the NIH Cur- 
riculum Supplement Series. I must say that I agree with the testi- 
mony that you received last month relating to how difficult it is to 
develop high-quality instructional materials. This is a time-con- 
suming and an expensive process. I believe that we have avoided 
the pitfalls described by forming appropriate partnerships. In 
short, we know what we know, and we know what we don’t know. 
We have been able to combine the scientific insights of some of the 
world’s leading scientists, people like Dr. Anthony Fauci and Dr. 
Francis Collins, with the professional expertise of some of the most 
highly respected curriculum development organizations in this 
country. 

We are currently engaged in aligning our supplements to the 
science, math, health, and language arts standards for each state. 
We have included, in an online appendix, a sample of alignments 
for a number of sample states, and I have included for Committee 
Members some examples of curriculum supplements for you to re- 
view. We believe that this project is one way that we can bring 
some of the excitement, hope, and promise of NIH research to 
schools around the Nation. But they won’t help if they don’t get out 
there, so I would like to close with a brief discussion of dissemina- 
tion. 

We have worked very hard to let educators know about the avail- 
ability of our now 16 different curriculum supplements aimed at el- 
ementary, middle, and high school. At the end of May, we had had 
requests from more than 70,000 teachers from across the country, 
for almost 285,000 supplement titles. In the online appendix, I 
have included maps showing the number and location of these re- 
quests for some sample states. I think if you, each of you, would 
look at your home state, as I do, you can look simply from the re- 
quests, and identify various towns. 

I don’t have time to show you all these state maps, but I can give 
you, show you a national map that gives you a sense of this. This 
map places one blue dot in each zip code from which we have re- 
ceived one or more requests for an NIH curriculum supplement. 
Each blue dot could represent one curriculum supplement or hun- 
dreds, but the map does give you a good sense of where the orders 
have come from. 

The next map is actually a NOAA satellite image of the U.S. at 
night, with lights showing approximately the population distribu- 
tion across the U.S. Now, if I can successfully toggle this back and 
forth, I will give you some sense of how well we have done at find- 
ing people where they live and connecting them with our cur- 
riculum supplement project. 

Thank you very much for the opportunity to discuss a few of our 
STEM programs. I will be happy to answer any of the questions 
that the Committee Members might have. 

[The prepared statement of Dr. Fuchs follows:] 

Prepared Statement of Bruce A. Fuchs 

Chairman Baird and Members of the Committee, it is a privilege to accept your 
invitation to participate in this hearing and provide you with information about 
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STEM education efforts at the National Institutes of Health (NIH), an agency of the 
Department of Health and Human Services. 

The mission of the NIH is to uncover new knowledge that will lead to better 
health for everyone. NIH has long been involved in directing programs for the collec- 
tion, dissemination, and exchange of information in medicine and health, including 
the development and support of medical libraries and the training of medical librar- 
ians and other health information specialists. In 1991, the NIH formed an Office 
of Science Education Policy (now the Office of Science Education under the Office 
of Science Policy) in the Office of the Director because of concerns surrounding the 
state of science education in the Nation. 

The NIH Office of Science Education (OSE) coordinates a program to strengthen 
and enhance efforts of the NIH to attract young people to biomedical and behavioral 
science careers and to improve science literacy in both adults and children. The 
function of the OSE is to: 1) develop, support, and direct program activities at all 
levels, with special emphasis on targeting students in grades kindergarten to 16, 
their educators and parents, and the general public; 2) advise NIH leadership on 
science education issues; 3) examine and evaluate research and emerging trends in 
science education and literacy for policy-making; 4) work closely with the NIH extra- 
mural, intramural, women’s health, laboratory animal research, and minority pro- 
gram offices on science education special issues and programs to ensure coordination 
of NIH efforts; 5) work with NIH Institutes and Centers to enhance communication 
of science education activities; and 6) work cooperatively with other public- and pri- 
vate-sector organizations to develop and coordinate activities. 

NIH contributes to K-16 STEM education in three main ways: 1) by partnering 
with educators on high-quality model programs to create instructional materials, 
conduct teacher professional development, and support informal science education in 
museums and science centers; 2) by conducting rigorous research into science and 
mathematics learning and teaching through NIH’s National Institute of Child 
Health and Human Development (NICHD); and 3) because of its interactions with 
scientists and students from around the world, by helping to understand the “world 
class standards” our students will need to compete in today’s world. 

la. What steps has your agency taken to improve its coordination with 
other federal agencies’ STEM education activities? 

NIH was actively engaged in the Academic Competitiveness Council (ACC) delib- 
erations. NIH Director Elias Zerhouni joined the ACC at the invitation of Secretary 
Spellings and made clear his support for the process. NIH participated in all three 
ACC working groups: K-12, Graduate/Postgraduate, and Outreach and Informal 
Education. 

Dr. Zerhouni has also committed NIH to a leadership role on the new National 
Science and Technology Council (NSTC) Subcommittee on STEM Education that 
will follow through on the ACC recommendations. Dr. Zerhouni appointed Dr. 
Duane Alexander, NICHD Director, to serve as one of the co-chairs of the sub- 
committee, along with Dr. Cora Marrett, National Science Foundation, and Dr. Russ 
Whitehurst, Department of Education. 

Additionally, the NIH is in discussions with the Department of Defense and the 
National Aeronautics and Space Administration (NASA) about ways to extend the 
ACC database (ACC Recommendation #1). By expanding the program database to 
include project-level information, federal program managers with shared interests 
(e.g., teacher professional development) would be able to find one another in order 
to share information. 

lb. To what extent does your agency collaborate with educators in the 
states and school districts in developing STEM education programs? 

Some of the resources that teachers request from the NIH were not originally tar- 
geted for classroom use. Most of the large number of publications created by NIH 
are directed to specific health conditions or are directed at specific audiences, such 
as patients, family members, and health care professionals. However, once these 
publications were discovered by science teachers, they began to be requested for use 
in classrooms as well. (Two popular examples are of this type of publication are In- 
side the Cell, available at http: // publications.nigms.nih.gov / insidethecell / , and Un- 
derstanding the Immune System, at http : / / www.niaid.nih.gov / publications / im- 
mune/ the _ immune -system.pdf . ) 

However, today the majority of NIH programs and resources requested by edu- 
cators were created expressly for, and with, teachers. Below are two examples of 
NIH resources currently available to science educators. 

First, the NIH National Center for Research Resources Science Education Part- 
nership Award (SEPA) Grant Program is the largest single K-12 (and informal) 
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science education program at NIH. SEPA’s goals are: 1) to stimulate career opportu- 
nities in basic science and clinical research by providing inquiry-based curricula to 
K-12 students, teachers, and parents; and 2) through SEPA projects at science cen- 
ters and museums, to increase the public’s understanding of NIH-funded medical re- 
search and to provide information about healthy life style choices. Because these 
awards are made to a community organization, the projects can be specifically de- 
signed to meet the needs of that community. For examples of SEPA-funded projects 
in selected States, see Appendix A at http: // science.education.nih.gov / HSTC, and 
for additional information about SEPA, see http: / / www.ncrr.nih.gov / 
science - education -partnership -awards / . 

Second, OSE has collaborated with a number of NIH Institutes and Centers to 
create a series of free curriculum supplements (currently 16 titles) for science edu- 
cators (available at http: / / science.education.nih.gov / supplements). Teachers have 
input into the development, writing, and editing of each supplement. The supple- 
ments are field-tested by teachers across the Nation and modified to address their 
concerns before being released to the public. Our state-level collaborations have in- 
cluded working with state departments of education and state-wide education advo- 
cacy groups (in New York, North Carolina, Ohio, Missouri, and Tennessee) to deter- 
mine whether a supplement meets a need in the state-wide science curriculum and 
to help with state health education standards development. The North Carolina De- 
partment of Public Instruction has recommended one of the NIH supplements as a 
primary resource for their eighth-grade science teachers since 2005. (http :/ / 
www.dpi.state.nc.us/docs/curriculum/science/middlegrades/8thsciencesupport.pdf . ) 

2. The recent report of the Academic Competitiveness Council reinforces 
the need for better evaluation and performance metrics for federal 
STEM education programs. What plans does your agency have for im- 
provements in its evaluation of its STEM programs? 

NIH supports the ACC goal of conducting increasingly rigorous evaluations of its 
STEM education activities using multiple evaluation strategies. These strategies 
will include working toward conducting randomized controlled trials where appro- 
priate. 

NIH has agreed to align its goals and metrics to those defined through the ACC 
process. The first NIH-wide meeting of K-12 project directors was held in April 
2007 to discuss this alignment as well as ways to begin collaborating on increasingly 
rigorous evaluations. 

The science education grants programs at NIH (for example, SEPA grants) are 
currently considering changes in their funding opportunity announcements to re- 
quire increasingly rigorous project evaluations. While this process will take some 
time, OSE is committed to helping the community (extramural grantees and NIH 
intramural project managers) solve the problems it may encounter on the road to 
rigorous evaluations. 

3. The Subcommittee received testimony at a hearing on 15 May on how 
the R&D mission agencies could improve the effectiveness of their STEM 
education programs. The witnesses were skeptical of the ability of the 
agencies to develop curricular materials for formal classroom instruc- 
tion and questioned the effectiveness of their teacher professional devel- 
opment programs to improve teacher classroom performance, while sug- 
gesting that the agencies’ most important role is in informal STEM edu- 
cation. The witnesses also strongly recommend closer collaboration by 
the agencies with educators in the field when developing STEM pro- 
grams. What is your response to the recommendations from these wit- 
nesses? 

Several witnesses expressed concerns at the May 15 hearing regarding the poten- 
tial pitfalls related to developing curricular materials for formal classroom instruc- 
tion. Mr. Michael Lach made comments about the problems of “adding more topics 
to cover” and of parochial projects’ being “harder to connect to our work” in terms 
of curriculum materials. Dr. Nelson noted, “There is a huge inventory of poorly de- 
signed and under-evaluated mission-related curricula (posters and lesson plans and 
associated professional development) rarely used in classrooms and with no natural 
home in a coherent standards-based curriculum. Effective curriculum development 
requires a deep collaboration with a team of professional curriculum developers, 
education researchers, and classroom teachers.” 

We could not agree more. Development of high-quality instructional materials is 
a difficult, time-consuming, and expensive undertaking. It is true that well-meaning 
scientists have unwittingly added to Dr. Nelson’s “huge inventory of poorly designed 
and under-evaluated” curricula. We believe that OSE has avoided these pitfalls by 
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proceeding slowly, doing our homework, understanding where we have expertise, 
and, most importantly, understanding where we do not have expertise. 

Before starting the NIH Curriculum Supplements Series, we conducted nearly two 
years of research, discussions, and interviews with leading curriculum developers 
across the U.S. We also conducted focus groups with educators at a number of con- 
ferences around the Nation to determine whether there was interest in having NIH 
create supplemental materials for the classroom. (There was.) We discussed with 
educators the topic areas where they felt they needed help and how these might be 
fit into biology courses. Interestingly, the teachers also strongly warned us “not to 
let our scientists write the curricula,” advice that we took to heart. 

When considering what NIH can bring to the creation of supplemental instruc- 
tional materials, it is important to note that our employees include some of the 
world’s leading scientific minds. Dr. Anthony Fauci, Director of the National Insti- 
tute of Allergy and Infectious Diseases, and Dr. Francis Collins, Director of the Na- 
tional Human Genome Research Institute, are only two such individuals who have 
contributed their scientific understanding and foresight to the NIH Curriculum Sup- 
plements Series. However, while NIH has this kind of scientific expertise in abun- 
dance, we do not have in-house expertise in instructional materials development. 

Instead, we have sought out professional curriculum development organizations 
that are as well known and respected in their field of expertise as NIH is in its own. 
We have contracted with BSCS (Biological Sciences Curriculum Study) and EDC 
(Educational Development Corporation), two of the most highly respected science in- 
structional materials developers in the Nation. Both of these organizations rely on 
research into how children learn science, use professional curriculum developers, 
and depend on classroom teachers as advisors, writers, and field-testers. Both of 
these organizations trace their genesis back to the early post-Sputnik days and have 
established long track records of creating innovative and effective curricula. 

NIH curriculum supplements were designed from the start to align with the Na- 
tional Science Education Standards (NSES). Most States have used the NSES to 
create their own standards documents. Since implementation of the No Child Left 
Behind Act, alignment to the NSES is no longer sufficient. As a result, we are un- 
dertaking the task of aligning each of our 16 curriculum supplement titles to each 
state’s science, mathematics, health, and language arts standards (34 States and the 
District of Columbia are done so far; see Appendix B at http :/ / 
science.education.nih.gov/HSTC for samples of alignments of one supplement to se- 
lected States). When this project is complete, we will be able to demonstrate for each 
state how a specific NIH curriculum supplement directly addresses the science and 
cross-curricular content standards that educators are expected to cover. 

Many educators have reported being especially excited to receive materials that 
can transmit some of the thrill and sense of discovery arising from the latest NIH 
research as a way to inspire and motivate their students. Each supplement provides 
activities for students to investigate science content knowledge they can apply di- 
rectly to some aspect of their daily lives. The fact that the materials cover the bio- 
logical concepts that teachers are required to cover but do it through references to 
human health and disease is seen as a strong positive. For instance, in general, chil- 
dren do not get very excited by studying onion root tips. It is far more engaging 
to study the mechanisms that control cell growth by relating it to a human disease 
like cancer. 

In creating these instructional materials, we were also motivated by the fact that 
research into the poor performance of our students in international comparisons has 
concluded that curricula in the U.S. are “a mile wide and an inch deep” and that 
content is often years out of date. The American Association for the Advancement 
of Science Project 2061 has evaluated many middle and high school science text- 
books and found all of them wanting. None of the 10 evaluated high school biology 
textbooks received even a “good” rating. We were convinced that teachers would 
benefit from free, accurate, interesting, standards-based instructional materials that 
incorporate the latest research into how people learn, so we developed curriculum 
supplements that allow students to think like present-day researchers and engage 
in practical applications. 

The extent to which we have created a curriculum series that is of interest to edu- 
cators is indicated by the fact that as of late May 2007, more than 70,000 educators 
have requested almost 285,000 supplements across the Nation. We would like to em- 
phasize that each of these supplements has been shipped out in response to a specific 
request for that title coming in from an educator. In other words, each of these re- 
quests is a record of a positive action taken by an educator to come to our website, 
fill out a post card, send us an e-mail, etc. (See Appendix C at http:/ / 
science.education.nih.gov/HSTC for distribution maps showing how many, and from 
where, requests have come for selected States.) 
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We are also proud to report that NIH curriculum supplements are frequently used 
as exemplary instructional resources by university-based professors engaged in 
teaching future science teachers in “methods” courses. The middle school supple- 
ment Doing Science: The Process of Scientific Inquiry has been especially well re- 
ceived by this audience. To our knowledge, very few other entities have created edu- 
cational materials that are deemed so useful that they are requested both by STEM 
teachers and by the university-based professors who train them. 

We created the NIH curriculum supplements as models for how challenging con- 
tent can be combined with engaging, realistic situations to give students the oppor- 
tunity to think like scientists. For a report on how the instructional model underpin- 
ning the NIH curriculum supplements aligns with current research into how people 
learn, see Appendix D at http:/ / science.education.nih.gov / HSTC. 

Last, although we share an enthusiasm for informal science education, we are 
concerned by the inequities that would result if it were our only approach. Many 
educators, particularly those in small, rural, or impoverished urban school districts, 
cannot afford a field trip to a science center or museum, nor is every school district 
within driving distance of a museum, major university, or federal laboratory instal- 
lation. We must not forget those teachers and students who cannot, for financial or 
other reasons, travel to a wonderful science museum, or have a scientist visit the 
classroom. These teachers and students also deserve to have access to high-quality 
science experiences. 

I would like to illustrate this last point with a personal anecdote. I was one of 
those rare individuals who knew from early childhood that he wanted to be a sci- 
entist. Undoubtedly, this was due, at least in part, to the post-Sputnik efforts that 
allowed my parents to order me pictures of astronauts, rockets, and stars from 
NASA. However, living where I did in central Illinois, I was a senior in high school 
before I got to meet my first working scientist — after a four-hour bus ride to Ar- 
gonne National Laboratory. We have designed the NIH curriculum supplements to 
bring some of the excitement, promise, and hope of NIH research to any school — 
urban or rural, rich or poor, with the best laboratory facilities or none at all. 

4. How does your agency determine priorities for its K-16 STEM education 
portfolio? Has your agency’s balance of programs at graduate/post doc- 
toral, undergraduate, K-12, and informal education changed much over 
the past few years? Do you foresee a change in that balance in the fu- 
ture? 

Approximately 95 percent of the education activities (in dollar terms) that NIH 
submitted to the ACC inventory fell into the “Graduate/Postgraduate” category. NIH 
has no plans to change that balance. 

In the future, this priority setting will be more formal and coordinated. As pre- 
viously mentioned, NIH has agreed to align its goals and metrics to those defined 
through the ACC process. The first NIH-wide meeting of K-12 project directors was 
held in April 2007 to discuss this alignment as well as ways to begin collaborating 
on increasingly rigorous evaluations. 

5. How does your agency disseminate information about its STEM edu- 
cation programs? What organizations, both government and private, 
have you partnered with to reach educators in the field? 

OSE has created a web site specifically designed to help educators find NIH re- 
sources that meet their needs (http: I / science.education.nih.gov). We also responded 
to input from teachers regarding the ways that they search for materials (for exam- 
ple, by topic, by grade level, by resource format). As OSE identifies new NIH re- 
sources, it codes them using this scheme to facilitate easy retrieval by teachers. 

Shortly before our last web site redesign, we began using the evaluation services 
of the American Consumer Satisfaction Index (ACSI), which publishes an e-govern- 
ment Satisfaction Index. ACSI is a cross-industry measure of consumers’ satisfac- 
tion. It measures the performance of over 200 private-sector companies as well as 
many government agencies, using scores calculated on data gathered from voluntary 
online surveys of randomly selected site visitors. For the past few years, the OSE 
web site has been one of the top 10 sites in the entire government in terms of cus- 
tomer satisfaction. 

Since March 2000, site traffic has increased from 17,000 visitor sessions per 
month to well over 250,000 visitor sessions per month. Web pages viewed each 
month have increased over the same time period from 36,000 to almost 2.5 million. 

For the past two years, OSE coordinated an “NIH Research Zone” at National 
Science Teachers Association (NSTA) national conferences. This year, eight NIH In- 
stitutes and Centers joined OSE, along with the Society for Neuroscience and other 
organizations. This effort has been greatly appreciated by the NSTA members. 
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NSTA President Linda Froschauer cited it as a good example of how NSTA benefits 
from interactions with federal agencies in her May 15, 2007, testimony before this 
committee. OSE also attends the National Association of Biology Teachers meetings, 
the National Middle School Association meetings, and, on occasion, state meetings 
of science teachers. 

Thank you for this opportunity to discuss NIH’s STEM education efforts with you. 
I will be happy to answer any questions you may have. 

Biography for Bruce A. Fuchs 

Dr. Bruce A. Fuchs is currently the Director of the National Institutes of Health’s 
(NIH) Office of Science Education (OSE). Dr. Fuchs is responsible for monitoring a 
range of science education policy issues and providing advice to NIH leadership. He 
also directs the creation of a series of K-12 science education curriculum supple- 
ments that highlight the medical research findings of the NIH. The NIH Curriculum 
Supplement Series is designed to meet teacher’s educational goals as outlined in the 
National Science Education Standards and is available free to teachers across the 
Nation. The office also actively creates innovative science and career education web 
resources, such as the LifeWorks career exploration site, accessible to teachers and 
students across the Nation. These resources are available at http :/ / 
science, education, nih.gov. 

Dr. Fuchs is serving on the Education and Workforce Development Working 
Group of the National Science and Technology Council and on working groups of the 
Department of Education’s Academic Competitiveness Council. He was a member of 
the K-12 education focus group for the National Academy of Science’s report Rising 
Above the Gathering Storm, which was utilized in the Administration’s development 
of the American Competitiveness Initiative, which President Bush introduced in his 
2006 State of the Union address. In 2005, the Department of Education asked Dr. 
Fuchs to serve as the U.S. representative to the Asian Pacific Economic Cooperation 
meeting on Best Practices in Math and Science Education. For a number of years, 
Dr. Fuchs was the NIH representative to the Department of Education’s National 
Education Research Policy and Priorities Board. That experience led to his con- 
tinuing interest in the debate over how to make educational research more effective. 

Before coming to NIH, Dr. Fuchs — an immunologist who did research on the 
interaction between the brain and the immune system — was a researcher and teach- 
er on the faculty of the Medical College of Virginia. He had grant support from both 
NIH’s National Institute of Mental Health and the National Institute on Drug 
Abuse. He has a B.S. in Biology from the University of Illinois and a Ph.D. in Im- 
munology from Indiana State University. He was born and raised in Springfield, Il- 
linois. 


Discussion 

Mr. McNerney. Thank you, Dr. Fuchs. And I want to thank all 
the witnesses for making the effort. I know it is a challenge to 
come over here and prepare yourselves and all, and sit in front of 
this committee, so it is very good testimony. 

At this point, we will open our first round of questions, and the 
Chair recognizes himself for approximately five minutes, or maybe 
a little bit longer. 

I think the purpose of this hearing is really to give this com- 
mittee comfort in what is about to happen in STEM education, and 
I see it as this committee’s — that is about to be formed, or is al- 
ready formed — responsibility to make sure that this is executed in 
a way that takes advantage of the resources that are available. I 
think it is a large responsibility, so I applaud, again, all of you for 
wanting to participate. 

I see Dr. Marrett as the leader of this effort, rightly or wrongly, 
and so I am going to be picking on you a little bit more than the 
others perhaps, but there is no malicious intent here. Will the sub- 
committee be developing a strategic plan for federal STEM edu- 
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cation programs to help guide the priorities of the agencies? Is that 
something you see a part of your charter? 

Dr. Marrett. Yes, I do. This is an important question, because 
we are, at this point, developing the charter for the committee. And 
as I see it, yes, there has got to be, we must undertake some activi- 
ties that will strengthen, enhance what is taking place with agen- 
cies, that will recognize some of the distinctiveness, but will also 
work towards the level of coordination that is going to be expected. 

When you ask, then, should there be some impact, some sense 
of what the impact should be, we will certainly be looking for how 
to think about the entire enterprise, and what is the role of the in- 
dividual agencies within that? I am saying this, expressing to you 
our desire to have any suggestions, any questions and ideas, as we 
develop both the charter and the set of activities. 

Mr. McNerney. Well, will you be developing an annual plan, 
then, or some sort of an organization chart that we can look at? I 
know that prior to this meeting, the ACC report had catalogued the 
way that money is being spent in K-12 education. That is very im- 
portant, I think that is, in some sense, the relatively easy part of 
this task. We can find out where we are through a certain amount 
of research, and we can determine what our plans are. The real 
hard part is evaluating what we are going to get once we start 
doing it, and I was appreciating Dr. Winterton’s discussion of how 
some programs had worked, and they were exciting, and the kids 
got involved. 

It would be nice to have that sort of thing, not codified, but in 
some sort of description that people can refer to in a way that will 
be helpful. But then again, the question of evaluation, I think Dr. 
Valdez mentioned that we can evaluate programs after they are 
done, but the problem is that we need sort of a feedback mecha- 
nism, so that when a program is in progress, we can tell if it is 
working or not. So, the evaluation is really the most difficult as- 
pect, in my opinion, here. So, can we have a comment, Dr. Marrett? 

Dr. Marrett. Several comments. I will try to keep them brief. 

One of the things that you also heard during the testimony is the 
imperative for enhanced research, STEM education research, un- 
derstanding a lot more of what works, under what conditions, for 
whom, and through what processes. Since we know we have got to 
enhance that a lot more, and I am very encouraged when I hear 
my colleagues indicate what that will mean. That will have to play 
into the plans for evaluation. 

So, on the one hand, we will be proceeding, building on the 
knowledge that does exist about evaluation, having those conversa- 
tions of what are the appropriate kinds of measures, of indicators. 
How do we develop indicators that would be common enough across 
the agencies? We will do that, but we won’t, I won’t promise that 
we will have everything on an annual basis at the beginning, just 
because there are these very thorny but important questions about 
how do we build the knowledge base, the knowledge base that will 
be so important for the way the evaluation activities will have to 
take place. 

Mr. McNerney. Well, I like the sound of using the scientific 
method in evaluating this scientific education, but honestly, that is 
going to be our challenge. How do we know when we are sue- 
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ceeding? How do we make sure that the program is organized well? 
I think it is going to be important to see that plan to move forward. 

Mr. Valdez, a question, if I have time. I understand that you 
have been developing a strategic plan for the Department of En- 
ergy education programs, to ensure that they align with STEM 
workforce needs for the future. Would you tell the subcommittee 
the process you underwent to determine those educational needs, 
and how you used that information to create your strategic plan? 

Mr. Valdez. Yes, the strategic plan, as most strategic plans, is 
based on what our stakeholders think we should be doing, and 
aligning that to the mission of the Department of Energy. So, for 
example, we had a series of nine focus groups, involving 110 stake- 
holders, everything from the educational community to corporate 
America, to groups representing underrepresented populations, in 
a very structured way, talking about how the Department of En- 
ergy, which is a mission agency, could participate in STEM edu- 
cation and workforce development. 

And I think this gets to the heart of what this hearing is all 
about. Why would an agency like the Department of Energy be in- 
volved in this? And the reason is because we care about our future 
workforce. We support specific disciplines and specific areas of in- 
dustrial competitiveness that are important to the Nation, and as 
a result, we have to work with the educational community at all 
levels, not just K-12, but undergraduate, postdoc, graduate school, 
and then continuing lifelong learning. 

So, we developed a process where we brought in members of all 
those communities, to help us rationalize the resources that we 
have within the Department of Energy, to more effectively use 
them. And frankly, our most effective use is through the National 
Laboratories. You, Mr. Chairman, were at Sandia National Labs, 
and you know that these National Laboratories are embedded in 
the communities. They work with local school districts. They are 
the experts; and so, we deliver our products and services through 
them, and we try to align the strategic plan in a way that makes 
full and maximum use of our National Laboratories. 

Mr. McNerney. Thank you, Mr. Valdez. The Chair now recog- 
nizes Dr. Ehlers for five minutes. 

Mr. Ehlers. Thank you, Mr. Chairman. 

First of all, Dr. Fuchs, you mentioned as an old professor, you 
felt compelled to use PowerPoint. Let me point out, as a younger 
professor, that I think you did the right thing. I have read a num- 
ber of these articles about how terrible it is to use PowerPoint, and 
we shouldn’t use them so much, et cetera. I have decided all of 
those are written by people who don’t know how to prepare good 
slides. 

Dr. Marrett, first question for you. You described, you are going 
to reconstitute the NSTC, the National Science and Technology 
Council’s Subcommittee on Education and Workforce Development, 
for which I applaud you. It should never have been disbanded. You 
described the necessary qualifications as knowledge and experi- 
ence. 

So, would it also be helpful for you, in that process, to ensure 
that a minimum level of experience, or maybe the better word is 
stature within their agency, their seniority or whatever, be re- 
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quired, so that you can ensure that the appropriate individuals will 
be at the NSTC’s disposal through this means? And I think you 
know what I am getting at. I have nothing against junior per- 
sonnel. They usually have the best ideas, but you need some upper 
level people there to really make sure that this gets carried out, 
and that when someone on your subcommittee goes back home, and 
says well, we really should do this, that you don’t have some other 
person there who frowns and says, well, we really don’t have the 
money in the budget. But what is your response? 

Dr. Marrett. My response is that the way this is being set up 
is through the Committee on Science. The Committee on Science 
consists of the principals from the agencies. Hence, when we have 
asked for their nominees, the assumption is that whatever the level 
of the person, that person will have direct access to the top of the 
agency. You are quite right that it will be very important that 
there will be access to the centers of communication, to funding, 
but this is what we anticipate the direct, the process should help 
enhance that. 

Mr. Ehlers. Okay. I hope you are right. You may want to build 
something into the structure to ensure that. 

Next question, I know what the ranking of the budgets of the 
agencies are. Obviously, Dr. Fuchs, you have the richest agency, 
and then, we go down to NASA, then Energy, and then, finally, 
NSF. And we could have lots of arguments of how it should be. But 
I am curious how much is spent on education in each of your agen- 
cies. And we will go right to left, and ask Dr. Fuchs, what does 
NIH spend on these educational issues? What is your budget? 

Dr. Fuchs. I believe we submitted to the ACC around $800 mil- 
lion worth of activity, and that would have been in the year when 
our total budget was about $28 billion. 

Mr. Ehlers. Yeah. Okay. 

Dr. Fuchs. The majority, 95 percent of that, was actually grad- 
uate, postgraduate activity. 

Mr. Ehlers. Okay. And how much, any estimate how much is K— 
12 ? 

Dr. Fuchs. We submitted $36 million worth of K-12 activities, 
and about $6 million worth of informal education. 

Mr. Ehlers. Okay. Mr. Valdez. 

Mr. Valdez. My Office has a budget of $8 million per year. 
Through appropriated funds, we think the Department spends 
about $20 to $25 million per year on educational activities; and 
then, through the National Laboratories, both through overhead, 
other kinds of activities, partnerships with industry and other 
groups, there is probably about another $30 million that is spent 
by the National Laboratories. So that is a total of about $50 to $60 
million. 

Mr. Ehlers. Okay. And Dr. Winterton, I know NASA spends a 
huge amount of money on educational activities. What is yours? 

Dr. Winterton. Well, we do spend on education, in various 
places within NASA, so within the Office of Education, we plan on 
spending about spending about $153 million. Then, we also have 
each of our Mission Directorates, who also invest in education, be- 
cause it ties, again, directly back to their missions. 

Mr. Ehlers. Right. 
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Dr. Winterton. So, for the Science Mission, it is about $72 mil- 
lion, for Explorations, $4.8, Aeronautics, $2.8, and our Space Oper- 
ation Mission Directorate spends and supports activities directly of 
the centers, so if you want those specifics, I can provide that as 
well, so 

Mr. Ehlers. I think this gives me a general idea. And finally, 
Dr. Marrett, what is 

Dr. Marrett. Yes. For NSF, across the board, it is nearly $1 bil- 
lion, but this includes what is spent inside the Directorate for Edu- 
cation and Human Resources, as well as outside. So the Directorate 
itself has a budget of just over $700 million, but this is where I 
would again indicate that there are efforts, every part of the Foun- 
dation has education outreach public service activities that relate 
to education, and so, it comes to about nearly $1 billion. 

Mr. Ehlers. Okay. That is very helpful. It gives me some per- 
spective, and I may have to speak to Dr. Bodman or Mr. Rohr- 
abacher about pumping up your budget, Mr. Valdez. I really appre- 
ciate what you have done. I have encountered, in my experience in 
the schools, the K-12 schools, products of what you have done, and 
I think it is outstanding. 

The one part that bothers me, which I have tried to correct legis- 
latively, but so far have not had enough support, is the inability 
of teachers to locate what they may want or need at a particular 
time, and so what I have advocated is a clearinghouse, where all 
of the programs that you have available for schools would be listed, 
and this would be on a website, and any teacher who wanted to 
could access that, if they had a particular unit they were working 
on, and wanted to just look at what was available, that teacher 
could go on the website, type that in, the units would be displayed. 

And I really advocate an Amazon.com type of approach, where 
teachers who have used something would then evaluate, and just 
write a short evaluative statement, give a ranking from one to four, 
and this would be very useful. I hope that we can get that passed, 
because I think that would be immensely useful to the teachers, 
and would give much greater exposure to what you have done and 
what you have solved. 

With that, I yield back. 

Mr. McNerney. Thank you, Mr. Ehlers. The Chair now recog- 
nizes my good friend and mentor from Illinois, Mr. Lipinski. 

Mr. Lipinski. Thank you, Mr. Chairman. I would like to thank 
all of the witnesses for their testimony today, and the work that 
they are doing. I don’t know if there is anything really more impor- 
tant in ensuring America’s future than to improve our STEM edu- 
cation. I think it is very critical for our country. I have a degree 
in mechanical engineering, and I always point to, and many people 
have heard this now, my physics teacher when I was a junior in 
high school, who really is the one who inspired me to go ahead and 
go to college and major in mechanical engineering, really got me 
interested and excited in studying engineering. 

A couple of things that I wanted to ask about. First, Dr. Marrett, 
the new NSTC Subcommittee on STEM Education is a sub- 
committee of the NSTC Committee on Science. Do you think that 
this is a sufficiently high level organization to get the agencies’ at- 



227 


tention, or do you think there should be something at a higher 
level than that? 

Dr. Marrett. I am quite willing to try it at the level where it 
is. There was, in fact, a previous committee that did exist. The 
charter expired in last December, and so, it is not as if this is a 
brand new activity, and my colleagues, who were involved before, 
will know that we have some items that should come onto that 
agenda. I know there has been this question of what should be the 
level. I think that we can be effective when there is the level of 
conversation that is needed, the planning, wherever, as long as, as 
Chairman McNerney indicated, there is the strong connection to 
the top of agencies and to the conversations that must take place 
across and within agencies. But I am not bothered by the fact that 
this is being structured as a subcommittee of the Committee on 
Science. 

Mr. Lipinski. Now, back in November, I spoke to the National 
Science Board. They were in Chicago meeting, talking about STEM 
education, and right now, we are awaiting their report and rec- 
ommendation on what to do on STEM education. I understand it 
has been delayed now, until August. 

Dr. Marrett, or if any other witnesses have any, do you have any 
ideas about what is going to be coming out of this, or where do you 
see this going? 

Dr. Marrett. Well, let me first thank you very much for the 
presentation that you made, and in fact, I heard that it was an im- 
pressive set of comments that you gave. The National Science 
Board Committee is in the process of redrafting the report. That 
is why I am not sure my colleagues have seen what is still, for now, 
an internal document for the National Science Board, thus it would 
be difficult for me to describe where everything is likely to lead, ex- 
cept that I would say it, again, is likely to reinforce the imperative 
for our coordinated efforts, for our heightened attention for what 
must take place. So, to that extent, I see it as responding to and 
helping to clarify many of the issues this committee has dealt with, 
as well as the several reports that people have referred to. 

Mr. Lipinski. Well, I am very much looking forward to seeing 
that, as I know all of us are. In my last bit of time here, I wanted 
to ask Mr. Valdez about some of the programs that are going on 
with the DOE labs in helping with STEM education. I have talked 
to the Museum of Science and Industry in Chicago, who is working 
and has more plans to work with Argonne National Lab, to try to 
bring students down to the Museum to really get them interested, 
excited about these various areas. 

Now, a couple things that you talked about in your testimony. 
First, you describe activities that support K-12 teachers who are 
involved in research activities at the National Labs. Has there been 
an evaluation that DOE has done about the effectiveness of doing 
this? 

Mr. Valdez. We have not done a specific evaluation of that pro- 
gram. It is a relatively new program. It is a three year program. 
The first cadre of teachers completed their participation in the pro- 
gram just this past summer, and we have been developing the eval- 
uation techniques that will enable us to track what happens to 
them after they finish the program, and what they have learned 
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through the program. So, it is too soon to say whether or not we 
have evaluated it right. 

Mr. Lipinski. Well, I definitely want to support that. I think that 
is something we should be doing more of, and also, in addition, 
something you talked about was that Sandia National Lab, if the 
Chair will indulge me to ask one last question here, at Sandia Na- 
tional Lab, you mentioned how there is a pilot program for the Ad- 
ministration’s Adjunct Teacher Corps Initiative. Can you explain 
how this works? 

Mr. Valdez. Well, the Adjunct Teacher Corps has a goal of put- 
ting 30,000 adjunct teachers into classrooms by the year 2012 or 
2013, I forget which one, and the Department of Education came 
to the Department of Energy and said, “We don’t have the ability 
to recruit adjunct teachers the way that you do,” because adjunct 
teachers would come from industry, National Laboratories, working 
scientists, retired scientists. We were very excited about working 
with them, and Sandia stepped up to the plate, and decided they 
would pilot the concept. And so, over the next fiscal year, we are 
going to take a look at what Sandia does, and then see if we can 
model this to the other 17 National Laboratories in the DOE sys- 
tem. 

I think that illustrates what we do at the Department of Energy. 
We work, we design programs nationally, but we implement them 
locally through the National Laboratories, and through our other 
partners. And if you do that, even though our budget is small, Dr. 
Ehlers — and I do appreciate your voice of support for this — the fact 
of the matter is we reach a lot of people. You know, through our 
National Laboratories, we touch 250,000 K-12 students every year, 
and 19,000 K-12 educators. The National Laboratories use the 
money that we have at Department of Energy headquarters as seed 
money to then go out and work with industry, local school districts, 
and everybody else, to implement the programs, and it is a very ef- 
fective model, and one that I think could be useful for other federal 
agencies. 

Mr. Lipinski. Thank you. Thank you, Mr. Chairman. 

Mr. McNerney. Thank you, Mr. Lipinski. I believe the Com- 
mittee was interested in another round of questions, so I will recog- 
nize myself for another five minutes. 

Dr. Marrett, will the subcommittee develop and maintain a cata- 
logue of the federal STEM education programs, as the ACC did? I 
think that is important, at least having a quantifiable, a really 
quantifiable way to understand the connection and how money is 
being spent, whether there is overlap, waste, and so on. 

Dr. Marrett. Because the committee is still being put together, 
I have to hesitate to answer some questions about what the com- 
mittee will do, but this one, I can be confident about, because we 
have already had the conversations that said yes, the ACC report 
had, of course, had recommended this kind of continued compen- 
dium of programs, and that will be done. So, that part is going to 
certainly be continued, as related to the ACC process. 

I think it is likely, too, that we will come back to an issue that 
had been on the table earlier, and that is trying to understand 
what the very concepts mean in the programs. As my colleague Dr. 
Fuchs has said, that sometimes the term that is used to describe 
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a program need not mean the same thing from one place to an- 
other. So, as we try to develop a useful compendium of what takes 
place, that will mean looking very much at how the programs are 
defined to develop something that will be across the agencies, too, 
on that front. But yes, we will be a source of trying to provide that 
information about what takes place. 

Mr. McNerney. What about priorities? How are we going to go 
ahead, or how is the committee going to go ahead and determine 
what the priorities are, in terms of resource usage? 

Dr. Marrett. I am a bit hesitant to answer that one, on behalf 
of all of my colleagues from the committee yet to be established. 
I think I would say with some level of confidence we certainly will 
be looking at matters of priorities, paying attention to a lot of the 
information that is already around about what the Nation’s needs 
are for now and into the future, this cannot be just looking at what 
might have been appropriate some 20 years ago, but exactly how 
those will evolve is to be determined, through a process that we in- 
tend to be as engaging and engaged as possible. For that, I would 
say we would welcome any observations you might have. 

Mr. McNerney. Well, it was interesting to hear Dr. Winterton’s 
testimony, because she was talking about inspiring kids, and that 
is critically important, because no matter how many resources we 
offer, if the generation doesn’t accept the offer and get engaged in 
the process, then we are not going to get as far as we need to get. 

And I see some of our national challenges as being a tool for get- 
ting kids engaged, the threat of global warming, for example, a 
great national and international threat that cannot only show chal- 
lenge, in terms of risk, and the fear that that might bring, but also, 
the opportunity for intellectual achievement, for financial success, 
and so on. So some of these great national challenges ought to be 
involved in some way, if we can get that as a part of the program. 
I don’t know exactly how, but perhaps Dr. Valdez, you have, or Dr. 
Winterton, have a comment on that? 

Dr. Winterton. I think, especially, as we have the opportunity 
to engage faculty and students directly with our scientists, through 
our science missions, or through our space exploration, so students 
see the real world application, and we provide mentorship, so even 
in high school, students are working at Goddard Spaceflight Cen- 
ter, and actually doing research with our scientists. 

So, I think it is that really, understanding, you really have to be 
very good, not only when you work in our labs, but when you are 
working on the International Space Station, with international col- 
leagues, that they really apply themselves in their math and their 
science classes, that they start seeing a career for them within the 
aerospace industry, and understand that it is a great opportunity, 
that they see the kind of skills they need, and the competencies to 
be there. 

Mr. Valdez. I am a big fan of prioritizing things, and I think you 
need to go back to what are the core capabilities and the core mis- 
sions of the federal agencies, and in the case of the federal mission 
agencies, I think it really comes down to two things. 

First, we need to be supportive of the workforce and the missions 
that we have been entrusted to us by the Congress and by the U.S. 
taxpayer. And so, for the Department of Energy, it is support for 
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energy, environment, national security, and basic discovery science. 
And that helps inform what kinds of programs we should have for 
STEM education and STEM workforce. 

And then, we have a larger responsibility through the American 
Competitiveness Initiative, to support STEM education and literacy 
in general. And I think this is where your notion, Mr. Chairman, 
of having students become excited about science plays in, and agen- 
cies are uniquely placed to have students become excited, whether 
it is by going to a DOE National Laboratory, and seeing the Ad- 
vanced Photon Source, or riding the National Oceanographic and 
Atmospheric Administration’s ship, you know, an Antarctic ship. 

I mean, there are ways that we can do this, but we need to set 
those priorities, and rationalize the resources in the appropriate 
way. 

Mr. McNerney. Well, I agree with the idea, but that only 
reaches a fairly small segment of the population, I mean, in terms 
of children in inner city schools, I mean, they are not going to get 
out to the labs. We need to find a way to reach out and show them 
the national need, the national priority, and the need for them to 
get engaged for our nation’s future, and that is something that the 
Committee might consider in its future debates. 

Dr. Marrett. Well, I would say that the Committee will need to 
look at a number of options, because what is also been represented 
a bit here, and even more so, with some of our agencies not at the 
table, is that we work to enhance the level of excitement, the moti- 
vation, through a number of informal processes, through the activi- 
ties with Boy Scouts, for example, with community groups. One is 
not constrained to the very formal kinds of settings, and that is 
where we are also very interested in what is appropriate for the 
other kinds of settings, because we do not intend to leave behind 
segments of the population we know that must be important for 
where the Nation moves. 

Mr. McNerney. Well, my time has expired. I am going to recog- 
nize Mr. Ehlers. 

Mr. Ehlers. Thank you, Mr. Chairman. 

First of all, let me just make a comment, Mr. Chairman. I have 
been through many hearings over the years, and I have been work- 
ing on improving math, science education in the K-12 system for 
close to 50 years now, but especially in the last 14, in the Congress. 
This is the biggest and best audience we have ever had for this, 
so I think, Mr. Chairman, we are finally getting the word out, and 
people are interested in the topic. I find that very heartening. 

I know that the National Science Foundation declares education 
and public outreach as part of research proposals that are sub- 
mitted. Is that common in your other agencies as well, when inves- 
tigators submit a request for proposal? Do they have to also include 
a statement about the educational aspects that will come out of 
that? 

Dr. Fuchs. It is not a feature in the NIH. It was considered for 
a while back in the 1990s, under the previous Director, and the de- 
cision was not to make that similar requirement, so we don’t have 
that. 

Mr. Ehlers. Okay. Mr. Valdez. 
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Mr. Valdez. No, it is not part of the Department of Energy’s 
grant programs. 

Mr. Ehlers. Okay. And Dr. Winterton. 

Dr. Winterton. It is part of our Science Mission Directorate, so 
each of their principal investigators with their science missions are 
required to do an education and public outreach plan and imple- 
ment that. 

Mr. Ehlers. And the two agencies that do this, is there the pro- 
posal of how they are going to implement that education and out- 
reach? Is that considered in deciding whether or not to give the 
grant, or is that just sort of an automatic add-on? 

Dr. Marrett. No, for NSF, it is critical for what takes place, be- 
cause we have got the criteria of intellectual merit and broader im- 
pacts, and in that context of broader impacts, this is where, gen- 
erally, the outreach efforts, the education activities, are to be un- 
dertaken. Those, then, become critical in the decisions made about 
the awarding of grants. We wrote this process, trying to think of 
some ways to make sure that we can evaluate effectively the out- 
comes from those efforts. 

Mr. Ehlers. Okay. So, if someone submits a proposal for an ex- 
periment to measure Einstein’s effect in general relativity, that, 
the educational part, is considered as well as the scientific part. Is 
that true in NASA also, Dr. Winterton? 

Dr. Winterton. It is a really critical part of their plan, and we 
work closely, and offer our assistance in the Office of Education, to 
assist in educational outreach and dissemination of those opportu- 
nities as well. 

Mr. Ehlers. Okay. And I would just encourage Energy and NIH 
to consider doing the same thing. It is a very useful thing. I also 
just, Dr. Winterton, you mentioned something about a mathe- 
matical exercise relating to flying. Do you use flight simulators at 
all your programs for the schools? 

Dr. Winterton. We do have 

Mr. Ehlers. Commercially available flight simulators. 

Dr. Winterton. We provide those opportunities through our cen- 
ters. Now, the type of simulation the students were doing, and I 
know you are a budding pilot, and you might want to try this simu- 
lator, to make sure we are getting the right mission control ex- 
perts, is 

Mr. Ehlers. I have already flown a Shuttle. 

Dr. Winterton. — is a computer-based simulation, so that part is 
available at any school. 

Mr. Ehlers. Yeah. 

Dr. Winterton. So, we like that combination of affordable, read- 
ily available, but you can also go to a center to do the simulation 
as well. 

Mr. Ehlers. Okay. 

Dr. Winterton. And I think through some of our university and 
our space grants, that certainly is another opportunity to provide 
that in a broader sense. 

Mr. Ehlers. Okay. And I can’t give commercials, but I am fas- 
cinated with the potentials of the X-Plane program, because it ac- 
tually allows students to construct their own airplane, and you 
have to be fairly sophisticated, at least junior high, perhaps higher, 
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but it is an incredibly good educational experience, and the pro- 
gram is $50 or something like that, so any school could afford to 
have that. 

Well, thank you, Mr. Chairman. This has been an excellent hear- 
ing. The only missing component, and I don’t want to criticize, even 
though I should, since I am part of the minority, and that is our 
job, but there is one other agency that spends a huge amount of 
money on K-12 education, and that is the Defense Department. It 
is a smaller fraction of their budget than any of these agencies, but 
we might want to hear from them some time, too, about what they 
are doing. 

And they, incidentally, one plus factor of the Defense Depart- 
ment is their Army Schools, which are located around the world — 
I shouldn’t say Army — the schools are for their employees’ chil- 
dren — consistently rank higher than the average American school, 
particularly in the science and technology, and also, often equal the 
countries, the record of the countries that they are in. So, obvi- 
ously, the Defense Department is doing something right in their 
schools, and we might be interested in talking to them about that 
some time. 

With that, I will conclude, and thank you again, Mr. Chairman, 
for having this hearing. 

Mr. McNerney. Well, the Chair thanks the Ranking Member for 
his thoughtful comments, and I think that is something that we 
would want to move forward with. 

And before closing, I would like to thank the witnesses for, again, 
spending their afternoon, and also, the members of the audience. 
It is a nice day out there, so it is appreciated that you would come 
in here to hear our testimony. 

And the record will remain open for additional questions that 
Subcommittee Members may ask of the witnesses, so be prepared 
for that. 

And the witnesses are excused, and the hearing is now ad- 
journed. 

[Whereupon, at 4:00 p.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Cora B. Marrett, Assistant Director, Education and Human Resources 
Directorate, National Science Foundation 

Questions submitted by Representative Eddie Bernice Johnson 

Ql. Dr. Marrett, the new National Science and Technology Council (NSTC) sub- 
committee on STEM education is a subcommittee of the NSTC Committee on 
Science. Is this of a sufficiently high level organizationally to get the agencies’ 
attention ? Because of the widely recognized importance of STEM education, 
shouldn’t this be constituted as a new, independent NSTC committee ? 

Al. The membership of the STEM Education Subcommittee has now been estab- 
lished. The timely response from the agencies to the request for names and the re- 
sponsibilities of the persons designated suggests that the subcommittee structure 
can be effective. The persons selected to represent their agencies have sufficient 
oversight for personnel and budgets to act on behalf of their organizations. 

There are substantive reasons for an arrangement that connects STEM education 
to the larger Committee on Science (COS). First, the agenda on STEM education 
must reflect current developments in the wider realm of STEM research and devel- 
opment. The connection to the COS, the body charged with the broader world of re- 
search and development, helps ensure that the priorities for STEM education are 
aligned with the directions at the federal level of science, technology, engineering, 
and mathematics. Second, within the agencies STEM education is integrated into 
and not isolated from the objectives agencies have identified for science and engi- 
neering. The subcommittee structure, then replicates the pattern that agencies em- 
ploy and have found to be beneficial. 

The structure does not signal that the subcommittee is of secondary importance 
in the NSTC complex. The principals for the Committee on Science regard STEM 
education as of profound significance for the Nation and consequently anticipate 
regular exchanges about the work of the subcommittee. They, in turn, are com- 
mitted to facilitating and participating in exchanges between the subcommittee and 
any other relevant NSTC bodies. Likewise, the co-chairs of the subcommittee have 
ready access to the principals and aim to work cooperatively with them. In sum, 
then, the subcommittee has obtained from the agencies and the Committee on 
Science the support it will need to be an effective force in the pursuit of excellence 
for education in science, technology, engineering, and mathematics. 

Q2. The Academic Competitiveness Council (ACC) has recommended that no funding 
should be increased for a STEM education program “unless a plan for rigorous, 
independent evaluation is in place,” which is defined in the report as being Ran- 
domized Controlled Trials or, when that is not feasible, Well-Matched Compari- 
son Group Studies. One of our witnesses at the May 15th hearing, Dr. Weiss, 
stated that this approach would not be practical for the majority of programs 
in the federal R&D mission agencies. I would like to ask our witnesses to re- 
spond to Dr. Weiss’ opinion. Is this a practical way of evaluating programs at 
NASA, Energy, and NIH? What other types of evaluation methods might be bet- 
ter suited ? 

A2. The Academic Competitiveness Council (ACC) report does indeed recommend no 
additional funding to STEM education programs, in the absence of plans for rig- 
orous, independent evaluation. The report points out, however, that “no single de- 
sign or evaluation methodology is appropriate for all education studies,” and “the 
appropriate methodology should be selected based on the maturity of the activity” 
(p. 13). Furthermore, it acknowledges that programs can be arrayed along a con- 
tinuum that starts “generally with small-scale studies to test new ideas and gen- 
erate hypotheses, leading to increasingly larger and more rigorous studies to test 
the effects of a given intervention or activity on a variety of students and in a vari- 
ety of settings” (p. 13). Thus, the report does not advocate for the premature use 
of experimental or quasi-experimental methods to determine causality. It is also im- 
portant to remember as well that the ACC discussion is centered on measuring the 
impact of an educational activity on student outcomes (p. 15), not educational dif- 
ferences at the institutional or system level. 

The National Science Foundation recognizes the importance of using randomized 
controlled trials (RCTs) to establish cause-and-effect relationships between edu- 
cation programs and student outcomes. As Dr. Weiss stated, and the ACC report 
supports, RCTs and well-matched comparison group studies are not always feasible 
and applicable, nonetheless. For short-term outreach efforts or efforts to enhance 
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the institutional structure for STEM education, some activities cannot be standard- 
ized and controlled. For projects federal agencies and others undertake, the random- 
ized assignment of participants is not always feasible, practical, or ethical. Thus, the 
use of RCTs is appropriate depending on the program design and the research ques- 
tion. The ACC report recognizes this point; Dr. Weiss sought especially to reinforce 
it. 

Policy-making bodies frequently draw on evidence from multi-year studies that 
use mixed methods. What seems significant for the making of policy is the rigor of 
the approach that is taken, coupled with attention to the question or problem that 
is central to analysis. 
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Answers to Post-Hearing Questions 

Responses by Joyce L. Winterton, Assistant Administrator, Office of Education, Na- 
tional Aeronautics and Space Administration (NASA) 

Questions submitted by Representative Eddie Bernice Johnson 

Ql. The Academic Competitiveness Council (ACC) has recommended that no funding 
should be increased for a STEM education program “unless a plan for rigorous, 
independent evaluation is in place” which is defined in the report as being Ran- 
domized Controlled Trials or, when that is not feasible, Well-Matched Compari- 
son Group Studies. One of our witnesses at the May 15, 2007, hearing, Dr. 
Weiss, stated that this approach would not be practical for the majority of pro- 
grams in the federal R&D mission agencies. I would like to ask our witnesses 
to respond to Dr. Weiss’ opinion. Is this a practical way of evaluating programs 
at NASA, Energy, and NIH? What other types of evaluation methods might be 
better suited? 

Al. Establishment of standards for the evaluation of Federal STEM education pro- 
grams was extensively discussed within the ACC working groups. A hierarchy of 
study designs, weighted in favor of a research-oriented model focused on Random- 
ized Control Trials (RCTs), was adopted. However, several concerns were raised by 
NASA and other working group members. The main concern is that RCTs are pri- 
marily applicable to interventions, defined broadly by the education evaluation com- 
munity, as projects with highly specific features that target a precisely defined audi- 
ence in order to achieve a specific outcome under controlled, standardized condi- 
tions. There are a variety of other reasons why RCT models are not practical for 
some of NASA’s portfolio, including: (a) schools are not typically prepared to match 
control and treatment groups; (b) RCTs are complex and costly to properly imple- 
ment; and, (c) some of NASA’s education projects are not specific interventions ac- 
cording to the above definition but are instead designed to enhance the capabilities 
of the education and outreach community (e.g., the NASA Explorer Institutes project 
seeks to enhance the ability of science centers and museums to use NASA’s unique 
resources). 

NASA is in full agreement that sound, rigorous evaluations should be imple- 
mented to provide the best possible evidence of effectiveness but the methodologies 
must be appropriate to the program. NASA supports and is implementing a defini- 
tion of rigorous evaluation that includes RCT-based methodologies, as feasible. 
Where RCTs are not feasible, NASA plans to implement alternate evaluation meth- 
odologies, commonly called “Mixed Methods” approaches, to capture a complete pic- 
ture of education investments to determine effectiveness in achieving outcomes, im- 
pact and comprehensiveness. These Mixed Methods are based on the accepted pro- 
fessional standards for educational evaluations articulated in The Program Evalua- 
tion Standards: How to Assess Evaluations of Educational Programs, 2nd edition, 
(Joint Committee on Educational Evaluation, 1994). Like RCTs, these rigorous 
methods will lead to credible, objective, reliable, and valid evaluations of program 
performance and effectiveness. Mixed-methods evaluations use quantitative data, 
such as experimental, quasi-experimental, and correlational studies, and also use 
qualitative methodologies, such as case studies, surveys, and focus groups. NASA 
will use RCTs when practical and relevant, however the Agency will typically use 
mixed-methods evaluations. 

NASA has developed and submitted to the Office of Management and Budget a 
plan for evaluating each of the Agency’s major projects. 
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Answers to Post-Hearing Questions 

Responses by William J. Valdez, Director, Office of Workforce Development for Teach- 
ers and Scientists, Office of Science, Department of Energy 

Questions submitted by Representative Eddie Bernice Johnson 
Federal STEM Education Programs 

Ql. The Academic Competitiveness Council (ACC) has recommended that no funding 
should be increased for a STEM education program “unless a plan for rigorous, 
independent evaluation is in place” which is defined in the report as being Ran- 
domized Controlled Trials or, when that is not feasible, Well-Matched Compari- 
son Group Studies. One of our witnesses at the May 15th hearing, Dr. Weiss, 
stated that this approach would not be practical for the majority of programs 
in the federal R&D mission agencies. I would like to ask our witnesses to re- 
spond to Dr. Weiss’ opinion. Is this a practical way of evaluating programs at 
NASA, Energy, and NIH? What other types of evaluation methods might be bet- 
ter suited ? 

Al. The Office of Science’s Workforce Development for Teachers and Scientists 
(WDTS) program fully supports the Academic Competitiveness Council (ACC) rec- 
ommendations and has developed a rigorous evaluation program currently under re- 
view by the Office of Management and Budget. That review program, which is being 
designed to ensure that the cost is commensurate with the overall size of the WDTS 
program, emphasizes three evaluation protocols that are consistent with the com- 
ments made by Dr. Weiss. Those three protocols, which we believe are appropriate 
to the needs of mission agencies such as NASA, the Department of Energy and NIH, 
are: 

• Quasi-experimental approaches that utilize well-matched groups to under- 
stand improvements to the learning of scientific content, STEM career 
choices, and improvements to STEM teaching approaches; 

• Long-range longitudinal studies of student participants designed to verify 
whether WDTS programs are achieving their goal of contributing to the DOE 
STEM workforce; and, 

• Management effectiveness studies, such as external reviews by Committees of 
Visitors, to validate that WDTS programs are managed efficiently. 

This evaluation program is designed for the mission needs of the Department of 
Energy, which requires development of a highly qualified pool of scientific and tech- 
nical workers in the mission areas it supports (defense, environment, energy, and 
scientific discovery). 
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Answers to Post-Hearing Questions 

Responses by Bruce A. Fuchs, Director, Office of Science Education, National Insti- 
tutes of Health 

Questions submitted by Representative Eddie Bernice Johnson 
Federal STEM Education Programs 

Ql. The Academic Competitiveness Council (ACC) has recommended that no funding 
should be increased for a STEM education program “unless a plan for rigorous, 
independent evaluation is in place” which is defined in the report as being Ran- 
domized Controlled Trials or, when that is not feasible, Well-Matched Compari- 
son Group Studies. One of our witnesses at the May 15th hearing, Dr. Weiss, 
stated that this approach would not be practical for the majority of programs 
in the federal R&D mission agencies. I would like to ask our witnesses to re- 
spond to Dr. Weiss’ opinion. Is this a practical way of evaluation programs at 
NASA, Energy, and NIH? What other types of evaluation methods might be bet- 
ter suited 1 

Al. It is important to articulate the final goal of federal STEM education efforts. 
We should seek to fund programs that really make a difference in the lives of our 
children. We want those students with talent and inclination to consider STEM ca- 
reers in order to help the government, and the Nation, meet its need for technical 
professionals. But additionally, in an increasingly competitive world, we want all 
students to obtain the knowledge and skills (for example, problem solving and crit- 
ical thinking) that they will need to find good jobs and lead fulfilling lives. The 
kinds of skills that students can learn in STEM courses will help prepare them for 
the 21st century — even if they never put on a white coat or work in a laboratory. 

The ACC report does not really define rigorous research as “Randomized Con- 
trolled Trials, or when that is not feasible, Well-Matched Comparison Group Stud- 
ies.” The report does emphasize the importance of these evaluation methods for 
“those study designs whose purpose is to estimate a project’s impact on education 
outcomes, such as student math and science achievement.” But the report also ac- 
knowledges that much of the federal STEM effort does not fall into this category. 
The report recognizes that “no single study design or evaluation methodology is ap- 
propriate for all education studies, and that the appropriate methodology should be 
selected based on the maturity of the activity, the intended use of the data, and the 
inferences to be drawn from study results. . 

Early in any STEM education research endeavor, most studies will not be can- 
didates for a randomized controlled trial design. These studies will more likely focus 
on classroom observations, developing early-stage instructional materials or ap- 
proaches, and generating testable hypotheses with regard to student performance. 
The ACC recognizes that these types of studies “are a key part of the research agen- 
da needed to improve U.S. STEM education, can be ‘rigorous’ in their own context, 
and can serve as valuable precursors and-or complements to impact studies.” 

However, the goal should ultimately be to determine what works for students. 
Promising programs at different stages of maturity should be identified and sub- 
jected to evaluation methods that have increasing power to discern whether the stu- 
dent outcomes observed are really related to the program interventions. This goal 
will push study designs up the “hierarchy pyramid” toward randomized controlled 
trials. However, even in these cases, a variety of evaluation methods will be used 
to “complement” the impact study. This is because we will seldom be satisfied to 
know simply whether a particular educational intervention works or not — we want 
to know why it succeeds, or fails, to achieve the intended objective. 

We do know more about how to effectively teach STEM subjects to students than 
we did a generation ago. However, the number of such insights that have been rig- 
orously tested and that are known to be valid for large numbers of students under 
a variety of conditions is much smaller than we would like it to be. Carefully apply- 
ing the recommendations of the ACC report will help ensure that we are not in the 
same situation a generation from now. 
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Statement of the Office of Education 
National Oceanic and Atmospheric Administration 
U.S. Department of Commerce 

Question 1: What steps have agencies taken to improve coordination with other fed- 
eral agencies’ STEM education activities ? To what extent do agencies collaborate 
with educators in the states and school districts in developing STEM education pro- 
grams ? 

The National Oceanic and Atmospheric Administration (NOAA) is leveraging sev- 
eral relationships established through the Academic Competitiveness Council (ACC) 
to foster improved coordination among Federal Science, Technology, Engineering 
and Mathematics (STEM) education activities. For example, such relationships in- 
clude: (1) the synthesis of independent evaluations of federal STEM activities (as 
previously demonstrated by the National Aeronautics and Space Administration); (2) 
providing advice to other agencies (e.g., the National Science Foundation) on how 
to develop and implement evaluations programs consistent with the ACC rec- 
ommendations and metrics; and (3) seeking external advice and guidance to inform 
the design of NOAA STEM education activities (e.g., from the Department of Edu- 
cation on NOAA’s Teacher at Sea Program). NOAA looks forward to exploring addi- 
tional opportunities through the National Science and Technology Council (NSTC) 
to build on the collaborative activities initiated under the ACC. 

NOAA is working with educators to develop and improve individual STEM edu- 
cation programs in many states and school districts. These programs are designed 
to support the long-term development of quality educational programs for all edu- 
cators and students while, simultaneously, meeting the goals of NOAA and the Na- 
tion. For example NOAA is working with California on the state’s Environmental 
Education Initiative, and NOAA is working with Hawaii to develop a Marine 
Science Curriculum. 

Question 2: The recent report of the Academic Competitiveness Council reinforces the 
need for better evaluation and performance metrics for federal STEM education pro- 
grams. What plans do agencies have for improvements in evaluation of STEM pro- 
grams ? 

NOAA is taking direct action to improve the consistency, rigor, and frequency of 
evaluation activities for each of its education programs. Specifically, NOAA will 
measure the effectiveness of all its current and future education activities using 
methodologies appropriate to the types of activities funded. NOAA has recently 
adopted a program logic model to inform the design and evaluation of all education 
activities in the agency, address the performance metrics of the Academic Competi- 
tiveness Council, and promote consistency, coordination, and information sharing 
with other federal entities. 

NOAA’s Educational Partnership Program (EPP) has adjusted its operation and 
implementation activities based on annual client evaluation; monitored the output 
and products of its programs against established performance measures; and as- 
sessed program statistics of students who have trained and/or graduated in STEM 
fields. In addition, NOAA’s Cooperative Science Centers, which are associated with 
EPP, were rigorously evaluated after three years of operation using metrics that 
were established previously to evaluate NOAA’s Cooperative Institutes. Evaluation 
metrics were tailored to address a variety of components such as education, out- 
reach, research, administration, recruitment, budgeting, and capacity building being 
conducted at each of the Cooperative Science Centers. The accomplishments of each 
program were compared to the objectives and performance measures established 
during the development phase. 

A plan to conduct rigorous, independent evaluation is currently being established 
for NOAA’s student scholarship programs, which include undergraduate and grad- 
uate fellowships, as well as the Ernest F. Hollings and Dr. Nancy Foster scholarship 
programs. Although these programs have not yet been formally evaluated, it should 
he noted that there are established performance measures in place for each pro- 
gram. 

Question 3: The Subcommittee received testimony at a hearing on 15 May on how 
the R&D mission agencies could improve the effectiveness of their STEM education 
programs. The witnesses were skeptical of the ability of the agencies to develop cur- 
ricular materials for formal classroom instruction and questioned the effectiveness of 
their teacher professional development programs to improve teacher classroom per- 
formance, while suggesting that the agencies’ most important role is in informal 
STEM education. The witnesses also strongly recommended closer collaboration by 
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the agencies with educators in the field when developing STEM programs. What are 
agencies’ responses to the recommendations from these witnesses? 

NOAA’s efforts do not replace or supplant the critical role of the State and local 
governments in education. Because there are limited resources to invest and large 
needs to support in science education, NOAA believes investments in formal and in- 
formal education are most effective and efficient at producing outcomes when they 
supplement or complement efforts supported by others. NOAA’s efforts are aimed 
at providing supplemental materials in subject areas where NOAA has unique ex- 
pertise and where sufficient public knowledge is required to ensure understanding 
and response to warnings, forecasts, and stewardship efforts. NOAA education 
maximizes use of place-based learning opportunities afforded by field offices, includ- 
ing National Marine Sanctuaries, Sea Grant Colleges, National Estuarine Research 
Reserves and Weather Forecast Offices. We often work through external partners, 
such as education associations or aquaria and science museums, to enhance our con- 
nections to the public and the education community. Students and educators are one 
of many groups of users of NOAA data and information. The NOAA Outreach Unit 
in Silver Spring, MD receives over 4,000 unique requests each year for NOAA’s edu- 
cation materials from teachers, students and librarians. 

NOAA recognizes that education is primarily the responsibility of state and local 
governments. However, state and local education programs may not specifically 
focus on topics relevant to NOAA’s mission. For example, the ocean sciences are 
under-represented in the national science education standards for grades K-12. In 
addition, teacher content knowledge is not always sufficient in the ocean and atmos- 
pheric sciences. As a result, students graduating from U.S. high schools may not 
possess sufficient understanding of the earth processes and phenomena that are the 
focus of NOAA research, monitoring, and prediction efforts. 

NOAA’s informal education activities provide educational experiences that typi- 
cally involve taking students to unique settings outside of the classroom. Informal 
education combines well-established educational methods with the excitement of 
hands-on activities and field experiences and develops life long interest in the ocean 
and atmosphere. NOAA’s informal education activities include hosting school chil- 
dren, community groups, and the general public at NOAA sites, supporting hands- 
on experiences in NOAA-related sciences and increasing the inclusion of NOAA-re- 
lated topics at science centers, museums, and aquaria. 

Question 4: How do agencies determine priorities for K-16 STEM education port- 
folio? Has agencies’ balance of programs at graduate I post doctoral, undergraduate, 
K-12, and informal education changed much over the past few years? Is there a like- 
lihood of a change in that balance in the future? 

Most of the investment priorities and the general direction of NOAA education ac- 
tivities over the past few years has been determined by Congress — either through 
legislation (e.g., the National Marine Sanctuaries Act, Coastal Zone Management 
Act, Sea Grant Act) or through appropriations language (Environmental Literacy 
Grants). Other education activities are supported by across the board percentages 
which Congress has instigated (e.g., NOAA-wide in the case of the Hollings Scholar- 
ship Program and Sanctuaries-wide in the case of the Nancy Foster Scholarship 
Program). Within these external constraints, NOAA’s process for determining prior- 
ities is based on national peer reviewed competitions focused on advancing earth 
system science education. Although there is no formal review process to determine 
priorities for funding, the following criteria are used informally: 

Proposed projects should: 

• deliver NOAA-wide benefit; 

• have hard schedule drivers that require action within a one to two year time 
frame; 

• reach a large, high priority audience; 

• result in a significant impact on the audience reached; 

• increase understanding of NOAA science and service; 

• leverage partnerships; 

• build on existing NOAA investments; and 

• target NOAA priority areas. 

Question 5: How do agencies disseminate information about STEM education pro- 
grams? What organizations, both government and private, have agencies partnered 
with to reach educators in the field? 
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NOAA relies on partnerships with a variety of government, non-government, non- 
profit, and private organizations to disseminate information about our STEM edu- 
cation programs. Partnerships mentioned above with State departments of edu- 
cation allow NOAA to directly assist state efforts to improve STEM-related edu- 
cation in areas specific to NOAA science (e.g., watershed and environmental edu- 
cation in the states that are party to the Chesapeake Bay agreement and its edu- 
cation commitments; environmental education in California; marine science edu- 
cation in Hawaii). 

NOAA also depends heavily on the dissemination networks of State and univer- 
sity partners for the education programs funded by the National Sea Grant College 
Program, the National Estuarine Research Reserve System, and cooperative re- 
search programs including the Educational Partnership Program, Joint Research In- 
stitutes, and Cooperative Institutes. 

Other federal partners, such as the National Aeronautics and Space Administra- 
tion and the National Science Foundation, leverage NOAA investments for broader 
impact including improving the rigor and credibility of high school Earth systems 
science course work and build centers for Coastal Ocean Science Educational Excel- 
lence. In the past few years, multiple partners joined NOAA to develop Ocean Lit- 
eracy Principles to assist teachers in using ocean concepts and examples to teach 
state science standards, and similar efforts are underway for climate literacy. NOAA 
also partners with the American Association for the Advancement of Science and the 
National Academy of Sciences to maintain the currency of the science education 
standards and benchmarks disseminated by these organizations. 

To reach classroom teachers directly, NOAA partners with organizations with 
large teacher membership and distribution networks and teacher training efforts 
(e.g., National Science Teacher Association, National Marine Educators Association, 
American Meteorological Society, the Jason Project). NOAA also supports online, 
searchable education resource libraries that provide access to standards-referenced, 
peer-reviewed education materials and lesson plans for teachers (e.g., Digital Li- 
brary for Earth Systems Science, www.dlese.org-, the Bridge, http:l / 
www2. vims, edu / bridge / noaa / ). 

Partnerships are essential to NOAA’s informal education efforts to promote 
STEM-related education related to NOAA science in the general public as well as 
supplementing the activities of the formal education system. NOAA’s partnerships 
with individual as well as networks of science centers, museums, aquaria, and zoos 
produce innovative exhibits and displays and complementary educational program- 
ming. These partnerships include world renowned institutions such as the American 
Museum of Natural History and the Smithsonian Institution National Museum of 
Natural History, and organizations such as American Zoo and Aquarium Associa- 
tion, Association of Science-Technical Centers, and Coastal Ecosystem Learning 
Centers). Partnerships with non-profit, private, and government organizations also 
enable NOAA’s efforts to improve public understanding related to STEM through 
radio and television programming and public media campaigns. 
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